Oct 30 21:14:00.760398 ip-10-0-128-26 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:14:00.760412 ip-10-0-128-26 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:14:00.760422 ip-10-0-128-26 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:14:00.760743 ip-10-0-128-26 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:14:10.982794 ip-10-0-128-26 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:14:10.982807 ip-10-0-128-26 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ed4f0a37cbef45c8ad90b45278863f94 -- Oct 30 21:16:09.438791 ip-10-0-128-26 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:16:09.838574 ip-10-0-128-26 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:09.838574 ip-10-0-128-26 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:16:09.838574 ip-10-0-128-26 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:09.838574 ip-10-0-128-26 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:09.845790 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845702 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:09.845790 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845787 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:09.845790 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845791 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:09.845790 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845794 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845797 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845800 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845803 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845806 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845808 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845811 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845815 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845819 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845823 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845826 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845829 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845831 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845834 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845836 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845851 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845854 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845856 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845859 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845861 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:09.845916 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845864 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845866 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845869 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845872 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845874 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845877 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845879 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845882 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845884 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845887 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845889 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845892 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845894 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845905 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845908 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845911 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845914 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845916 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845919 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845921 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:09.846374 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845924 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845927 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845930 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845932 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845935 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845937 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845940 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845943 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845951 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845954 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845957 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845961 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845963 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845966 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845968 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845971 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845974 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845976 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845979 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:09.846846 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845982 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845984 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845986 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845989 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845991 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845994 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.845998 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846001 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846003 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846006 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846008 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846011 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846013 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846016 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846018 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846021 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846023 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846026 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846029 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:09.847329 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846031 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846034 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846036 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846044 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846047 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846050 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846052 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846054 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846058 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846062 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846065 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846068 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846070 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846073 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846075 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846078 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846080 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846082 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846085 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:09.847956 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846088 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846091 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846094 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846096 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846100 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.846103 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847009 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847017 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847019 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847022 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847025 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847028 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847030 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847033 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847035 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847038 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847041 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847044 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847046 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847049 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:09.848605 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847052 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847054 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847057 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847060 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847063 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847065 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847068 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847071 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847073 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847076 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847078 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847082 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847084 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847087 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847089 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847093 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847095 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847098 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847100 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847103 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:09.849083 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847105 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847108 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847110 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847112 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847115 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847120 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847123 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847126 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847128 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847131 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847134 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847137 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847139 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847142 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847144 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847147 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847150 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847152 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847155 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:09.849575 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847157 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847160 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847163 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847166 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847171 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847174 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847176 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847179 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847181 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847184 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847186 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847189 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847191 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847194 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847196 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847199 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847202 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847205 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847209 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:09.850028 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847212 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847230 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847234 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847238 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847242 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847252 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847255 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847258 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847260 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847267 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847270 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847273 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847275 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847278 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847280 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847283 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847285 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847291 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847294 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:09.850533 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847296 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847299 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847301 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847304 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847306 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847309 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847311 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847314 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847316 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847319 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847321 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847324 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847326 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847328 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.847331 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848863 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848876 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848880 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848884 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848886 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:09.850982 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848889 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848892 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848894 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848897 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848900 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848903 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848906 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848909 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848913 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848918 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848921 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848924 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848926 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848929 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848932 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848935 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848938 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848940 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848943 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:09.851452 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848945 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848948 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848951 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848954 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848956 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848959 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848961 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848964 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848967 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848971 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848975 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848977 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848980 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848983 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848986 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848988 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848991 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848993 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848995 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:09.851907 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.848998 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849000 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849003 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849005 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849008 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849010 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849012 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849015 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849017 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849020 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849022 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849025 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849028 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849031 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849033 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849036 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849039 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849041 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849044 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:09.852339 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849046 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849049 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849052 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849055 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849058 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849060 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849063 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849066 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849069 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849071 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849074 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849076 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849079 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849081 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849084 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849086 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849089 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849092 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849094 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849097 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:09.852802 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849099 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849102 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849105 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849108 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849110 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849113 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849115 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849118 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849121 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849124 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849126 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849129 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849131 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849134 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849136 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849139 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849141 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849143 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849146 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:09.853276 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849148 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849151 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849154 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849156 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.849159 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.849989 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.849998 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850006 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850011 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850015 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850018 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850023 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850027 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850030 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850034 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850038 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850042 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850045 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850048 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850050 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850055 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850058 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850061 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:16:09.853734 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850065 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850067 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850071 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850074 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850077 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850081 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850084 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850087 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850091 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850093 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850096 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850099 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850102 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850105 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850110 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850113 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850116 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850119 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850122 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850125 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850129 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850132 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850135 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850138 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:16:09.854266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850142 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850146 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850149 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850152 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850155 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850159 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850162 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850165 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850168 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850171 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850174 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850177 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850181 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850184 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850187 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850190 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850194 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850197 2574 flags.go:64] FLAG: --help="false" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850200 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850203 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850206 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850209 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850212 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:16:09.854928 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850216 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850219 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850221 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850224 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850227 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850230 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850233 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850236 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850239 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850242 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850245 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850248 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850251 2574 flags.go:64] FLAG: --lock-file="" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850254 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850257 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850261 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850267 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850270 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850273 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850276 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850279 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850282 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850285 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850288 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850293 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:16:09.855598 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850296 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850300 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850304 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850307 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850309 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850312 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850315 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850318 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850321 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850328 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850331 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850334 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850337 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850340 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850343 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850346 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850350 2574 flags.go:64] FLAG: --port="10250" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850353 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850356 2574 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e852a63d36ab7aa5" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850360 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850363 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850366 2574 flags.go:64] FLAG: --register-node="true" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850369 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850372 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850375 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850378 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:16:09.856196 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850381 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850385 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850388 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850391 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850394 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850397 2574 flags.go:64] FLAG: --runonce="false" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850400 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850403 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850406 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850409 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850412 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850415 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850418 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850421 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850424 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850427 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850430 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850433 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850436 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850439 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850441 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850449 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850452 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850455 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850459 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:16:09.856821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850464 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850467 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850470 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850473 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850477 2574 flags.go:64] FLAG: --v="2" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850481 2574 flags.go:64] FLAG: --version="false" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850497 2574 flags.go:64] FLAG: --vmodule="" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850502 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850505 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850653 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850658 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850661 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850664 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850667 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850670 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850672 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850675 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850678 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850681 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850683 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850686 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850688 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:09.857393 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850691 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850694 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850696 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850698 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850701 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850704 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850709 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850711 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850714 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850716 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850720 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850723 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850726 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850728 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850731 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850734 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850737 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850739 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850742 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850744 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:09.857936 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850747 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850750 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850752 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850754 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850757 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850759 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850762 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850764 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850767 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850769 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850772 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850774 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850777 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850780 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850782 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850784 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850787 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850791 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850796 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:09.858403 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850799 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850802 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850805 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850809 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850811 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850815 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850817 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850820 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850822 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850825 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850828 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850830 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850833 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850836 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850840 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850843 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850846 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850848 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850851 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850853 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:09.858889 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850856 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850858 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850861 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850863 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850866 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850868 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850871 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850874 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850876 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850879 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850881 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850885 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850888 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850890 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850893 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850897 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850900 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850902 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850905 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:09.859396 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850908 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850911 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850913 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850916 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850919 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850921 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850924 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850927 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850929 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850932 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850934 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850937 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850939 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850942 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.850944 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.850969 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:09.859859 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851021 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851026 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851030 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851032 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851036 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851039 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851042 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851044 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851047 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851050 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851054 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851058 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851063 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851066 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851069 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851071 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851075 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851077 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851080 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:09.860233 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851083 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851085 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851088 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851090 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851092 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851095 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851098 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851102 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851104 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851107 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851110 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851112 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851114 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851117 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851119 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851122 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851124 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851126 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851129 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:09.860709 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851131 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851134 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851137 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851140 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851142 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851145 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851149 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851151 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851154 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851156 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851159 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851162 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851164 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851167 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851169 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851172 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851174 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851177 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851179 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:09.861182 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851182 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851184 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851186 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851188 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851191 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851193 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851196 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851198 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851201 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851205 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851207 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851210 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851212 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851214 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851217 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851219 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851222 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851224 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851227 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851231 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:09.861646 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851234 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851236 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851238 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851241 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851243 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851246 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851248 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851251 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851253 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851256 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851258 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851260 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851263 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851265 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851267 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851270 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851272 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851275 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851277 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851280 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:09.862121 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851282 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851286 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851289 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851292 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851294 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851296 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851299 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851301 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.851304 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.859045 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.859062 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.859074 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.859079 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859139 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859143 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859146 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859149 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859152 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859156 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859159 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:09.862727 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859162 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859164 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859167 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859170 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859172 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859175 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859178 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859181 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859184 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859186 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859189 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859191 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859194 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859196 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859199 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859201 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859203 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859206 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859208 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:09.863242 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859211 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859213 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859216 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859220 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859223 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859226 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859229 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859231 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859234 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859237 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859239 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859242 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859244 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859247 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859249 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859252 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859254 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859257 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859259 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859262 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:09.863692 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859265 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859267 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859270 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859272 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859275 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859277 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859280 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859282 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859285 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859287 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859290 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859292 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859295 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859297 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859300 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859303 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859310 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859314 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859316 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859319 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:09.864181 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859322 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859324 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859327 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859329 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859332 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859335 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859337 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859340 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859342 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859345 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859347 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859350 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859353 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859355 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859358 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859360 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859363 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859365 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859368 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859370 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:09.864716 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859373 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859375 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859378 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859380 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859383 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859385 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859388 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859391 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859395 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859399 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859403 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859405 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859408 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859411 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859414 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859416 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859419 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859421 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859425 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:09.865193 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859429 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.859456 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859643 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859649 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859653 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859656 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859658 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859661 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859663 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859666 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859669 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859671 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859674 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859676 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859679 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859682 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:09.865684 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859684 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859687 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859689 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859692 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859696 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859699 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859702 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859705 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859708 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859710 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859713 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859715 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859718 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859720 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859723 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859725 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859727 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859730 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859734 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:09.866069 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859738 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859740 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859743 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859745 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859748 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859750 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859754 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859757 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859760 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859763 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859766 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859768 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859771 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859773 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859776 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859778 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859781 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859784 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859787 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859790 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:09.866619 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859792 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859795 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859798 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859800 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859803 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859805 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859808 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859811 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859813 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859816 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859818 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859821 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859823 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859826 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859828 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859830 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859833 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859835 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859837 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859840 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:09.867078 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859842 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859845 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859847 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859850 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859852 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859855 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859857 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859859 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859862 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859865 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859868 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859871 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859874 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859876 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859886 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859890 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859892 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859895 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859898 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859900 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:09.867580 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859902 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859905 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859907 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859910 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859912 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859915 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859917 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859919 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859922 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859925 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859927 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859930 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859932 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.859955 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859987 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859991 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:09.868076 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859994 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.859997 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860000 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860002 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860005 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860008 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860011 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860013 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860016 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860018 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860021 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860024 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860026 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860029 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860031 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860034 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860036 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860039 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860042 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860044 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:09.868448 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860047 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860049 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860052 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860054 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860057 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860060 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860063 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860065 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860068 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860070 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860073 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860075 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860078 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860080 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860082 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860085 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860088 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860091 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860094 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860096 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:09.868939 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860099 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860102 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860105 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860107 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860110 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860113 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860115 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860118 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860120 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860123 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860126 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860128 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860130 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860133 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860135 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860138 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860140 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860143 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860145 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:09.869404 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860147 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860150 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860154 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860158 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860161 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860164 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860166 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860169 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860171 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860175 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860179 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860181 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860185 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860187 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860190 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860192 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860195 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860197 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860200 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:09.869872 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860203 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860205 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860208 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860210 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860213 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860215 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860218 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860220 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860222 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860225 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860227 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860230 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860234 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860236 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860239 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860242 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860245 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860248 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860250 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860253 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:09.870340 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860255 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860258 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860260 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860262 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860265 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:09.860267 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.860318 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.862836 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.863884 2574 server.go:1014] "Starting client certificate rotation" Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.863987 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.864019 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.865999 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:16:09.870873 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.868009 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:16:09.880234 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.880218 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:16:09.885025 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.885006 2574 log.go:25] "Validated CRI v1 image API" Oct 30 21:16:09.885878 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.885857 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:16:09.888446 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.888427 2574 fs.go:135] Filesystem UUIDs: map[15a7d64e-bc98-4a4f-a078-83a0e9867baa:/dev/nvme0n1p3 605eda00-e80d-442f-8dda-38a65156c993:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 21:16:09.888508 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.888446 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:16:09.894869 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.894752 2574 manager.go:217] Machine: {Timestamp:2026-10-30 21:16:09.894251491 +0000 UTC m=+0.350888863 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099323 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29285f5664f0eb5fd8d20b01857dc5 SystemUUID:ec29285f-5664-f0eb-5fd8-d20b01857dc5 BootID:ed4f0a37-cbef-45c8-ad90-b45278863f94 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:01:1c:c7:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:01:1c:c7:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:a0:af:6c:15:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:16:09.894869 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.894855 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:16:09.895033 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.894937 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:16:09.898364 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.898342 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:16:09.898574 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.898366 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-26.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:16:09.899269 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.899257 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:16:09.899301 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.899271 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:16:09.899301 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.899285 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:16:09.899301 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.899298 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:16:09.900561 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.900551 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:16:09.904936 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.904924 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:16:09.905056 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.905047 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:16:09.905881 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.905872 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:16:09.905917 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.905885 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:16:09.905917 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.905901 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:16:09.905917 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.905914 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:16:09.906018 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.905923 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:16:09.906310 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.906296 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:16:09.908690 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.908675 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:16:09.915331 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.915318 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:16:09.916249 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.916234 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:16:09.916284 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.916268 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:16:09.916429 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.916419 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:16:09.916476 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.916435 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:16:09.916476 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.916443 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:16:09.916476 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.916449 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:16:09.916476 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.916459 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:16:09.916476 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.916465 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:16:09.917148 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.917140 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:16:09.917176 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.917149 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:16:09.917176 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.917157 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:16:09.917176 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.917164 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:16:09.917254 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.917178 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:16:09.917254 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.917187 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:16:09.918957 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.918946 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:16:09.918957 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.918958 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:16:09.923021 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.923008 2574 server.go:1263] "Started kubelet" Oct 30 21:16:09.923238 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.923127 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:16:09.923238 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.923113 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:16:09.923363 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.923272 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:16:09.923998 ip-10-0-128-26 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:16:09.924857 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.924840 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:16:09.926032 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.926015 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:16:09.928464 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:09.928445 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:16:09.933034 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.933018 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:16:09.933120 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:09.933096 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" Oct 30 21:16:09.933120 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.933019 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:16:09.933754 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.933734 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:16:09.933754 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.933739 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:16:09.933883 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.933759 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:16:09.933883 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.933881 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:16:09.933984 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.933893 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:16:09.934205 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.934053 2574 factory.go:55] Registering systemd factory Oct 30 21:16:09.934205 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.934070 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:16:09.934319 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.934306 2574 factory.go:153] Registering CRI-O factory Oct 30 21:16:09.934364 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.934321 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 21:16:09.934414 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.934373 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:16:09.934414 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.934388 2574 factory.go:103] Registering Raw factory Oct 30 21:16:09.934414 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.934399 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 21:16:09.935938 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.935920 2574 manager.go:319] Starting recovery of all containers Oct 30 21:16:09.942857 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.942822 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:16:09.949114 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.949095 2574 manager.go:324] Recovery completed Oct 30 21:16:09.953003 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.952990 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:09.955482 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.955461 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:09.955585 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.955549 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:09.955585 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.955561 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:09.956167 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.956148 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:09.956167 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.956152 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:16:09.956246 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.956164 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:16:09.956246 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.956210 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:16:09.956910 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.956895 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:09.956972 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.956924 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:09.956972 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.956934 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:09.958437 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.958424 2574 policy_none.go:50] "Start" Oct 30 21:16:09.958518 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.958442 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:16:09.958518 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.958453 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:16:09.960547 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.960538 2574 policy_none.go:44] "Start" Oct 30 21:16:10.000046 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:09.999884 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:16:10.017886 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.017861 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 21:16:10.018570 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.018554 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:16:10.018631 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.018573 2574 server.go:87] "Starting device plugin registration server" Oct 30 21:16:10.018849 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.018835 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:16:10.018905 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.018852 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:16:10.019005 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.018982 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:16:10.019114 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.019055 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:16:10.019114 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.019063 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:16:10.019591 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.019573 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:16:10.019647 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.019618 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" Oct 30 21:16:10.066754 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.066715 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-26.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:16:10.068793 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.066727 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-26.us-west-1.compute.internal.18e36b392c3b151d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-26.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-26.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:16:09.922966813 +0000 UTC m=+0.379604185,LastTimestamp:2026-10-30 21:16:09.922966813 +0000 UTC m=+0.379604185,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-26.us-west-1.compute.internal,}" Oct 30 21:16:10.079971 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.079949 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:16:10.079971 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.079975 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:16:10.080128 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.079994 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:16:10.080128 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.080028 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:16:10.116522 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.116429 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-26.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:16:10.119078 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.119064 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:10.119832 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.119818 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:10.119916 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.119851 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:10.119916 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.119867 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:10.119916 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.119892 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.138597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.138578 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:16:10.138697 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.138622 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r5s8c" Oct 30 21:16:10.180912 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.180873 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal"] Oct 30 21:16:10.181049 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.180947 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:10.181850 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.181834 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:10.181938 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.181863 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:10.181938 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.181872 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:10.184100 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184088 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:10.184244 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.184297 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184262 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:10.184842 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184825 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:10.184935 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184852 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:10.184935 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184866 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:10.184935 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184890 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:10.184935 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184910 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:10.184935 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.184922 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:10.186983 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.186966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.187068 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.186999 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:10.187726 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.187707 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:10.187801 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.187734 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:10.187801 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.187744 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:10.215965 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.215947 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.218453 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.218433 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.221877 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.221862 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:10.235956 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.235933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4207178a950303c256d4f333088d96f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"a4207178a950303c256d4f333088d96f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.236035 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.235960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.236035 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.235977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.320937 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.320907 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:10.322219 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.322201 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:10.322456 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.322439 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:10.322864 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.322849 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.322903 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.322872 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-26.us-west-1.compute.internal\": node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" Oct 30 21:16:10.336266 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.336243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4207178a950303c256d4f333088d96f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"a4207178a950303c256d4f333088d96f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.336327 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.336273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.336327 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.336298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.336419 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.336350 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.336419 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.336353 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4207178a950303c256d4f333088d96f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"a4207178a950303c256d4f333088d96f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.336419 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.336365 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.433573 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.433478 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.496373 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.496341 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:16:10.496552 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.496452 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.518695 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.518673 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.521829 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.521807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:10.551137 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.551110 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:16:10.863302 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.863230 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:16:10.863877 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.863380 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:16:10.863877 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.863408 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:16:10.863877 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:10.863453 2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a307518d15e3f4f029c52010fdce43ee-4fb9e135dd735df7.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-26.us-west-1.compute.internal.18e36b392e2c7e60\": read tcp 10.0.128.26:54780->18.225.247.67:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-26.us-west-1.compute.internal.18e36b392e2c7e60 default 4977 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-26.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-26.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-26.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:16:09 +0000 UTC,LastTimestamp:2026-10-30 21:16:10.119870925 +0000 UTC m=+0.576508297,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-26.us-west-1.compute.internal,}" Oct 30 21:16:10.907135 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.907106 2574 apiserver.go:52] "Watching apiserver" Oct 30 21:16:10.933667 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:10.933638 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:16:11.143517 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.143410 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:11:10 +0000 UTC" deadline="2028-04-09 02:05:01.963829643 +0000 UTC" Oct 30 21:16:11.143517 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.143444 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12628h48m50.82038846s" Oct 30 21:16:11.167138 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.167115 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:16:11.167436 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.167419 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-55bl6","openshift-ovn-kubernetes/ovnkube-node-sk6xz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n","openshift-multus/multus-kd826","openshift-network-diagnostics/network-check-target-79bgk","openshift-network-operator/iptables-alerter-6r5rg","openshift-cluster-node-tuning-operator/tuned-n7bs2","openshift-multus/multus-additional-cni-plugins-84tpc","kube-system/global-pull-secret-syncer-bbwl8","kube-system/konnectivity-agent-tljvj","openshift-dns/node-resolver-zpmnj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal","openshift-image-registry/node-ca-xlrvq"] Oct 30 21:16:11.169931 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.169908 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:11.170056 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.169978 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:11.172080 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.172068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.174223 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.174206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.176285 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.176270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kd826" Oct 30 21:16:11.178400 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.178382 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:11.178468 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.178449 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:11.180567 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.180552 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.182874 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.182858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.184923 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.184908 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.187099 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.187086 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.187434 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.187408 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:11.190287 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.190272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:11.192466 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.192453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.194754 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.194739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.226647 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.226614 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-msgtv\"" Oct 30 21:16:11.226647 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.226614 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:16:11.228114 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.228101 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrcct\"" Oct 30 21:16:11.228592 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.228581 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:16:11.229104 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.229091 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ks46b\"" Oct 30 21:16:11.229521 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.229482 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:16:11.229748 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.229733 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:16:11.229809 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.229759 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:16:11.229851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.229821 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:16:11.229890 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.229870 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:16:11.229974 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.229963 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:16:11.230009 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.229966 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:16:11.230182 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.230169 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:16:11.230316 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.230303 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:16:11.230355 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.230313 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:16:11.230355 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.230342 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:16:11.232137 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.232127 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:16:11.234547 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.234532 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:16:11.241134 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241116 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31857e99-9c08-45ea-85da-f154ca58223d-iptables-alerter-script\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.241212 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5btq\" (UniqueName: \"kubernetes.io/projected/31857e99-9c08-45ea-85da-f154ca58223d-kube-api-access-f5btq\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.241212 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-k8s-cni-cncf-io\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.241212 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-daemon-config\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.241361 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgk6\" (UniqueName: \"kubernetes.io/projected/7b8cf76d-e722-49b5-8c62-fe88eff85691-kube-api-access-6fgk6\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.241361 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241266 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.241361 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241301 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-conf\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.241361 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-lib-modules\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.241548 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-hosts-file\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.241548 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241394 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-registration-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.241548 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-device-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.241548 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:11.241548 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-etc-kubernetes\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.241548 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-tmp\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.241548 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmmrz\" (UniqueName: \"kubernetes.io/projected/cb67e001-cef7-4f5e-9f38-2617953250d4-kube-api-access-rmmrz\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-netns\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e226384-4104-4da9-a6b9-ce21038fedc3-ovn-node-metrics-cert\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-socket-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241625 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-cnibin\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysconfig\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-env-overrides\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-hostroot\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.241744 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-sys\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241750 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-systemd-units\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-node-log\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xc7r\" (UniqueName: \"kubernetes.io/projected/2e226384-4104-4da9-a6b9-ce21038fedc3-kube-api-access-7xc7r\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-netns\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-run\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-slash\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-system-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-cnibin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vg5\" (UniqueName: \"kubernetes.io/projected/6df5ee35-2300-4a44-9ca3-c40545ab2866-kube-api-access-48vg5\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:11.241995 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.241985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-etc-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-sys-fs\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-multus\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62b9feff-59c5-45d5-a398-c78ad8dd590b-agent-certs\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-systemd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-script-lib\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-modprobe-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-kubelet-config\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-kubelet\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242208 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242253 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-bin\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-binary-copy\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.242372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-cni-binary-copy\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-bin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ghs\" (UniqueName: \"kubernetes.io/projected/66b958bb-61d6-473c-bf4d-21fefe6e8675-kube-api-access-55ghs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-tuned\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242447 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62b9feff-59c5-45d5-a398-c78ad8dd590b-konnectivity-ca\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jv87\" (UniqueName: \"kubernetes.io/projected/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-kube-api-access-2jv87\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-var-lib-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-log-socket\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242548 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-netd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-config\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31857e99-9c08-45ea-85da-f154ca58223d-host-slash\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-system-cni-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-conf-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-kubernetes\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-dbus\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb67e001-cef7-4f5e-9f38-2617953250d4-host\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.242888 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-ovn\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242734 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242749 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgc2k\" (UniqueName: \"kubernetes.io/projected/06b8bac2-6f8c-4a6a-9953-0573489e7846-kube-api-access-vgc2k\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242763 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-socket-dir-parent\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242776 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-kubelet\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-systemd\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-tmp-dir\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb67e001-cef7-4f5e-9f38-2617953250d4-serviceca\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-multus-certs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-host\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kube-api-access-mkwc6\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.243386 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.242996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-var-lib-kubelet\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.269145 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.269125 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:16:11.269238 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.269149 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jmnhg\"" Oct 30 21:16:11.269375 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.269357 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:16:11.269375 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.269367 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:16:11.269481 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.269408 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:16:11.271347 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.271333 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kknvl\"" Oct 30 21:16:11.276293 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.276275 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fp6d9\"" Oct 30 21:16:11.277826 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.277812 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8g8v4\"" Oct 30 21:16:11.278074 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.278061 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:16:11.278125 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.278062 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:16:11.278125 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.278066 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:16:11.278959 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.278945 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:16:11.279000 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.278949 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:16:11.279037 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.279028 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:16:11.279658 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.279643 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:16:11.279742 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.279728 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ztz2s\"" Oct 30 21:16:11.279813 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.279802 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:16:11.279986 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.279975 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:16:11.280085 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.280074 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6llv\"" Oct 30 21:16:11.280137 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.280087 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:16:11.280187 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.280152 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l4xz7" Oct 30 21:16:11.344334 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.343611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31857e99-9c08-45ea-85da-f154ca58223d-iptables-alerter-script\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.344334 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344247 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31857e99-9c08-45ea-85da-f154ca58223d-iptables-alerter-script\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5btq\" (UniqueName: \"kubernetes.io/projected/31857e99-9c08-45ea-85da-f154ca58223d-kube-api-access-f5btq\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-k8s-cni-cncf-io\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-daemon-config\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgk6\" (UniqueName: \"kubernetes.io/projected/7b8cf76d-e722-49b5-8c62-fe88eff85691-kube-api-access-6fgk6\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-conf\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-lib-modules\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-hosts-file\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-registration-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-device-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-etc-kubernetes\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.344986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-tmp\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmmrz\" (UniqueName: \"kubernetes.io/projected/cb67e001-cef7-4f5e-9f38-2617953250d4-kube-api-access-rmmrz\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-netns\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e226384-4104-4da9-a6b9-ce21038fedc3-ovn-node-metrics-cert\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.345712 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-socket-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-cnibin\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345174 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysconfig\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-env-overrides\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-hostroot\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-sys\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-systemd-units\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-node-log\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xc7r\" (UniqueName: \"kubernetes.io/projected/2e226384-4104-4da9-a6b9-ce21038fedc3-kube-api-access-7xc7r\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-netns\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-run\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-slash\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-system-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345554 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-cnibin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345604 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:11.346645 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vg5\" (UniqueName: \"kubernetes.io/projected/6df5ee35-2300-4a44-9ca3-c40545ab2866-kube-api-access-48vg5\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-etc-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345753 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysconfig\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345766 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345813 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-etc-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345869 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-etc-kubernetes\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345905 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-system-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-sys-fs\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-multus\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62b9feff-59c5-45d5-a398-c78ad8dd590b-agent-certs\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-systemd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-script-lib\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346080 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346144 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345756 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-node-log\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.345873 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-socket-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346346 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-lib-modules\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346478 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.347267 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346511 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-conf\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346561 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-sys-fs\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346566 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-k8s-cni-cncf-io\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346564 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-sys\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346623 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-hosts-file\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346623 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-cnibin\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346610 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-registration-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346661 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-multus\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346666 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-netns\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-modprobe-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346672 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-daemon-config\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-kubelet-config\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346735 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-kubelet-config\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346797 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-modprobe-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346809 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-cnibin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346826 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-hostroot\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.346919 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-systemd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347411 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-systemd-units\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348054 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347636 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-netns\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347747 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-device-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347760 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-run\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347775 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-env-overrides\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347808 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-slash\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-kubelet\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.347897 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347934 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347956 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-script-lib\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.347968 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs podName:6df5ee35-2300-4a44-9ca3-c40545ab2866 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:11.847947504 +0000 UTC m=+2.304584885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs") pod "network-metrics-daemon-55bl6" (UID: "6df5ee35-2300-4a44-9ca3-c40545ab2866") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347986 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-kubelet\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.347993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348032 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-bin\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-binary-copy\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348093 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.348930 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348123 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-bin\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-cni-binary-copy\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348183 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-bin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ghs\" (UniqueName: \"kubernetes.io/projected/66b958bb-61d6-473c-bf4d-21fefe6e8675-kube-api-access-55ghs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-tuned\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62b9feff-59c5-45d5-a398-c78ad8dd590b-konnectivity-ca\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jv87\" (UniqueName: \"kubernetes.io/projected/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-kube-api-access-2jv87\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-var-lib-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-log-socket\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-netd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-config\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31857e99-9c08-45ea-85da-f154ca58223d-host-slash\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-system-cni-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-conf-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-kubernetes\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-dbus\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.349851 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb67e001-cef7-4f5e-9f38-2617953250d4-host\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-ovn\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348562 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-binary-copy\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgc2k\" (UniqueName: \"kubernetes.io/projected/06b8bac2-6f8c-4a6a-9953-0573489e7846-kube-api-access-vgc2k\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348628 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-socket-dir-parent\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348679 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-socket-dir-parent\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348688 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-kubelet\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-systemd\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348720 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-tmp-dir\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb67e001-cef7-4f5e-9f38-2617953250d4-serviceca\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348763 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-netd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-multus-certs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-host\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.350597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kube-api-access-mkwc6\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.348923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-var-lib-kubelet\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349008 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-var-lib-kubelet\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349150 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-tmp-dir\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349208 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-config\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349231 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349272 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-kubelet\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349277 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-system-cni-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349301 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-cni-binary-copy\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349319 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-conf-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349305 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31857e99-9c08-45ea-85da-f154ca58223d-host-slash\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349337 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-systemd\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349371 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-bin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349379 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-dbus\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349382 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb67e001-cef7-4f5e-9f38-2617953250d4-host\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349427 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-ovn\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349431 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb67e001-cef7-4f5e-9f38-2617953250d4-serviceca\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.351103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349428 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-var-lib-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349517 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349558 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-multus-certs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349610 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-host\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349608 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-kubernetes\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.349657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-log-socket\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.349689 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.349745 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret podName:db28a65b-b841-4446-b062-9526c7e8139a nodeName:}" failed. No retries permitted until 2026-10-30 21:16:11.849728656 +0000 UTC m=+2.306366020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret") pod "global-pull-secret-syncer-bbwl8" (UID: "db28a65b-b841-4446-b062-9526c7e8139a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.350247 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-tmp\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.350580 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62b9feff-59c5-45d5-a398-c78ad8dd590b-agent-certs\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.350622 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e226384-4104-4da9-a6b9-ce21038fedc3-ovn-node-metrics-cert\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.351649 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.350858 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62b9feff-59c5-45d5-a398-c78ad8dd590b-konnectivity-ca\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:11.351980 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.351898 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-tuned\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.401966 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.401875 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:11.401966 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.401910 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:11.401966 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.401926 2574 projected.go:196] Error preparing data for projected volume kube-api-access-q7ccw for pod openshift-network-diagnostics/network-check-target-79bgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:11.402375 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.401999 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw podName:4401a9ff-fa39-4669-b5b1-3c9f473904cf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:11.901974753 +0000 UTC m=+2.358612131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q7ccw" (UniqueName: "kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw") pod "network-check-target-79bgk" (UID: "4401a9ff-fa39-4669-b5b1-3c9f473904cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:11.403922 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.403896 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5btq\" (UniqueName: \"kubernetes.io/projected/31857e99-9c08-45ea-85da-f154ca58223d-kube-api-access-f5btq\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.410846 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.410821 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgk6\" (UniqueName: \"kubernetes.io/projected/7b8cf76d-e722-49b5-8c62-fe88eff85691-kube-api-access-6fgk6\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.410967 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.410947 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xc7r\" (UniqueName: \"kubernetes.io/projected/2e226384-4104-4da9-a6b9-ce21038fedc3-kube-api-access-7xc7r\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.411037 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.410965 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmmrz\" (UniqueName: \"kubernetes.io/projected/cb67e001-cef7-4f5e-9f38-2617953250d4-kube-api-access-rmmrz\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.411037 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.411011 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vg5\" (UniqueName: \"kubernetes.io/projected/6df5ee35-2300-4a44-9ca3-c40545ab2866-kube-api-access-48vg5\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:11.413155 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.412790 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgc2k\" (UniqueName: \"kubernetes.io/projected/06b8bac2-6f8c-4a6a-9953-0573489e7846-kube-api-access-vgc2k\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.413155 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.412854 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kube-api-access-mkwc6\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.413155 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.413112 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jv87\" (UniqueName: \"kubernetes.io/projected/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-kube-api-access-2jv87\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.413354 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.413282 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ghs\" (UniqueName: \"kubernetes.io/projected/66b958bb-61d6-473c-bf4d-21fefe6e8675-kube-api-access-55ghs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:11.519562 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.519531 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:11.525882 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.525864 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:11.540619 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.540589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kd826" Oct 30 21:16:11.558162 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.558131 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:11.560512 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.560483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:11.585980 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.585950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:11.592742 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.592719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:11.613749 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.613724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:11.617786 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.617762 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:11.649016 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:11.648979 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4207178a950303c256d4f333088d96f.slice/crio-a6f2b267e9e8df3ff3c62cd817c7221faad572c9bd8cb7033a6cfea47f48e99f WatchSource:0}: Error finding container a6f2b267e9e8df3ff3c62cd817c7221faad572c9bd8cb7033a6cfea47f48e99f: Status 404 returned error can't find the container with id a6f2b267e9e8df3ff3c62cd817c7221faad572c9bd8cb7033a6cfea47f48e99f Oct 30 21:16:11.654333 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:11.654309 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8b937f14ab0703a8793557804f6cae.slice/crio-edd5b9c34cf497f9ceb39247a4185820f4b18eae4ad2b7e08f259df9e10e6911 WatchSource:0}: Error finding container edd5b9c34cf497f9ceb39247a4185820f4b18eae4ad2b7e08f259df9e10e6911: Status 404 returned error can't find the container with id edd5b9c34cf497f9ceb39247a4185820f4b18eae4ad2b7e08f259df9e10e6911 Oct 30 21:16:11.655068 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.655043 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:16:11.852417 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.852375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:11.852612 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.852453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:11.852612 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.852535 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:11.852612 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.852575 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:11.852612 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.852603 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs podName:6df5ee35-2300-4a44-9ca3-c40545ab2866 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:12.852586573 +0000 UTC m=+3.309223934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs") pod "network-metrics-daemon-55bl6" (UID: "6df5ee35-2300-4a44-9ca3-c40545ab2866") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:11.852760 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.852622 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret podName:db28a65b-b841-4446-b062-9526c7e8139a nodeName:}" failed. No retries permitted until 2026-10-30 21:16:12.852609363 +0000 UTC m=+3.309246725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret") pod "global-pull-secret-syncer-bbwl8" (UID: "db28a65b-b841-4446-b062-9526c7e8139a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:11.952908 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:11.952795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:11.953459 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.952950 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:11.953459 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.952973 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:11.953459 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.952983 2574 projected.go:196] Error preparing data for projected volume kube-api-access-q7ccw for pod openshift-network-diagnostics/network-check-target-79bgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:11.953459 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:11.953036 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw podName:4401a9ff-fa39-4669-b5b1-3c9f473904cf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:12.953021804 +0000 UTC m=+3.409659165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7ccw" (UniqueName: "kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw") pod "network-check-target-79bgk" (UID: "4401a9ff-fa39-4669-b5b1-3c9f473904cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:12.084361 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.084298 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"2e8b937f14ab0703a8793557804f6cae","Type":"ContainerStarted","Data":"edd5b9c34cf497f9ceb39247a4185820f4b18eae4ad2b7e08f259df9e10e6911"} Oct 30 21:16:12.085380 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.085359 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"a4207178a950303c256d4f333088d96f","Type":"ContainerStarted","Data":"a6f2b267e9e8df3ff3c62cd817c7221faad572c9bd8cb7033a6cfea47f48e99f"} Oct 30 21:16:12.111993 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.111969 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:12.283579 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.283466 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:11:11 +0000 UTC" deadline="2028-04-22 10:46:27.231565933 +0000 UTC" Oct 30 21:16:12.283579 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.283517 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12949h30m14.948052284s" Oct 30 21:16:12.336394 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.336366 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:12.384849 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:12.384822 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e226384_4104_4da9_a6b9_ce21038fedc3.slice/crio-8658ed5d19dc364dbc4245355d250145efff810ecb022d42245d51ef0b9bd69d WatchSource:0}: Error finding container 8658ed5d19dc364dbc4245355d250145efff810ecb022d42245d51ef0b9bd69d: Status 404 returned error can't find the container with id 8658ed5d19dc364dbc4245355d250145efff810ecb022d42245d51ef0b9bd69d Oct 30 21:16:12.506226 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:12.506181 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9e20d3_04ff_4a1c_a5e2_4accc5aedd98.slice/crio-1c8cb6d9b415aabf60731948b82bea105b919fc29a9c6ae5ac813a6586b431d8 WatchSource:0}: Error finding container 1c8cb6d9b415aabf60731948b82bea105b919fc29a9c6ae5ac813a6586b431d8: Status 404 returned error can't find the container with id 1c8cb6d9b415aabf60731948b82bea105b919fc29a9c6ae5ac813a6586b431d8 Oct 30 21:16:12.523000 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:12.522974 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f30bd6_0ef2_47f0_8f55_6f5370a098e1.slice/crio-398dacbe0c53fe2584bd4dbf1f1394a67f46504d05cc4aaf1d8bb52a2aac077d WatchSource:0}: Error finding container 398dacbe0c53fe2584bd4dbf1f1394a67f46504d05cc4aaf1d8bb52a2aac077d: Status 404 returned error can't find the container with id 398dacbe0c53fe2584bd4dbf1f1394a67f46504d05cc4aaf1d8bb52a2aac077d Oct 30 21:16:12.530662 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:12.530597 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b8bac2_6f8c_4a6a_9953_0573489e7846.slice/crio-f01ec999358031a5ae5f89a39cb6047b2ed173353dd4a06e4580a9414e46ae84 WatchSource:0}: Error finding container f01ec999358031a5ae5f89a39cb6047b2ed173353dd4a06e4580a9414e46ae84: Status 404 returned error can't find the container with id f01ec999358031a5ae5f89a39cb6047b2ed173353dd4a06e4580a9414e46ae84 Oct 30 21:16:12.533494 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:12.533469 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b9feff_59c5_45d5_a398_c78ad8dd590b.slice/crio-9ed777e703955cf5d60a910190303922cd5d70cc37a0808176865b0fb10a3ee1 WatchSource:0}: Error finding container 9ed777e703955cf5d60a910190303922cd5d70cc37a0808176865b0fb10a3ee1: Status 404 returned error can't find the container with id 9ed777e703955cf5d60a910190303922cd5d70cc37a0808176865b0fb10a3ee1 Oct 30 21:16:12.544368 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:12.544344 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb67e001_cef7_4f5e_9f38_2617953250d4.slice/crio-6b3e40ef4b5de1ce1c7e5c1e98c1a391bb2c6efbd8d59d1374f607ae009c0c39 WatchSource:0}: Error finding container 6b3e40ef4b5de1ce1c7e5c1e98c1a391bb2c6efbd8d59d1374f607ae009c0c39: Status 404 returned error can't find the container with id 6b3e40ef4b5de1ce1c7e5c1e98c1a391bb2c6efbd8d59d1374f607ae009c0c39 Oct 30 21:16:12.570303 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:12.570275 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b958bb_61d6_473c_bf4d_21fefe6e8675.slice/crio-46616890441e21b0b9e09f556ee9ae630a204ed29c83a1efc962c83cfbca0057 WatchSource:0}: Error finding container 46616890441e21b0b9e09f556ee9ae630a204ed29c83a1efc962c83cfbca0057: Status 404 returned error can't find the container with id 46616890441e21b0b9e09f556ee9ae630a204ed29c83a1efc962c83cfbca0057 Oct 30 21:16:12.571744 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:12.571640 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31857e99_9c08_45ea_85da_f154ca58223d.slice/crio-7e22c0449cb818634227da175e2b41f8d8ab4b1ce82f8342c076e115ea1e86d5 WatchSource:0}: Error finding container 7e22c0449cb818634227da175e2b41f8d8ab4b1ce82f8342c076e115ea1e86d5: Status 404 returned error can't find the container with id 7e22c0449cb818634227da175e2b41f8d8ab4b1ce82f8342c076e115ea1e86d5 Oct 30 21:16:12.860820 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.860729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:12.860999 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.860817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:12.860999 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:12.860952 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:12.861133 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:12.861013 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs podName:6df5ee35-2300-4a44-9ca3-c40545ab2866 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:14.860993815 +0000 UTC m=+5.317631175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs") pod "network-metrics-daemon-55bl6" (UID: "6df5ee35-2300-4a44-9ca3-c40545ab2866") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:12.861133 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:12.861081 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:12.861133 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:12.861118 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret podName:db28a65b-b841-4446-b062-9526c7e8139a nodeName:}" failed. No retries permitted until 2026-10-30 21:16:14.861106247 +0000 UTC m=+5.317743610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret") pod "global-pull-secret-syncer-bbwl8" (UID: "db28a65b-b841-4446-b062-9526c7e8139a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:12.962048 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:12.962001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:12.962728 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:12.962206 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:12.962728 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:12.962228 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:12.962728 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:12.962243 2574 projected.go:196] Error preparing data for projected volume kube-api-access-q7ccw for pod openshift-network-diagnostics/network-check-target-79bgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:12.962728 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:12.962309 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw podName:4401a9ff-fa39-4669-b5b1-3c9f473904cf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:14.962289552 +0000 UTC m=+5.418926913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7ccw" (UniqueName: "kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw") pod "network-check-target-79bgk" (UID: "4401a9ff-fa39-4669-b5b1-3c9f473904cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:13.080507 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.080454 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:13.080699 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.080531 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:13.080699 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.080552 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:13.080699 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:13.080661 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:13.080851 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:13.080756 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:13.080851 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:13.080819 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:13.088720 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.088682 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kd826" event={"ID":"66b958bb-61d6-473c-bf4d-21fefe6e8675","Type":"ContainerStarted","Data":"46616890441e21b0b9e09f556ee9ae630a204ed29c83a1efc962c83cfbca0057"} Oct 30 21:16:13.090163 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.090131 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpmnj" event={"ID":"82f30bd6-0ef2-47f0-8f55-6f5370a098e1","Type":"ContainerStarted","Data":"398dacbe0c53fe2584bd4dbf1f1394a67f46504d05cc4aaf1d8bb52a2aac077d"} Oct 30 21:16:13.091622 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.091502 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" event={"ID":"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98","Type":"ContainerStarted","Data":"1c8cb6d9b415aabf60731948b82bea105b919fc29a9c6ae5ac813a6586b431d8"} Oct 30 21:16:13.093091 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.093063 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" event={"ID":"7b8cf76d-e722-49b5-8c62-fe88eff85691","Type":"ContainerStarted","Data":"cf4b3ac9f18f9170b5b7f99d0b0e47a713e799e01daabbb7ac117f0351be9a39"} Oct 30 21:16:13.094876 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.094851 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6r5rg" event={"ID":"31857e99-9c08-45ea-85da-f154ca58223d","Type":"ContainerStarted","Data":"7e22c0449cb818634227da175e2b41f8d8ab4b1ce82f8342c076e115ea1e86d5"} Oct 30 21:16:13.097466 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.097431 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xlrvq" event={"ID":"cb67e001-cef7-4f5e-9f38-2617953250d4","Type":"ContainerStarted","Data":"6b3e40ef4b5de1ce1c7e5c1e98c1a391bb2c6efbd8d59d1374f607ae009c0c39"} Oct 30 21:16:13.098801 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.098774 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tljvj" event={"ID":"62b9feff-59c5-45d5-a398-c78ad8dd590b","Type":"ContainerStarted","Data":"9ed777e703955cf5d60a910190303922cd5d70cc37a0808176865b0fb10a3ee1"} Oct 30 21:16:13.100107 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.100082 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84tpc" event={"ID":"06b8bac2-6f8c-4a6a-9953-0573489e7846","Type":"ContainerStarted","Data":"f01ec999358031a5ae5f89a39cb6047b2ed173353dd4a06e4580a9414e46ae84"} Oct 30 21:16:13.101749 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.101717 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"8658ed5d19dc364dbc4245355d250145efff810ecb022d42245d51ef0b9bd69d"} Oct 30 21:16:13.283812 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.283693 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:11:11 +0000 UTC" deadline="2028-03-27 20:24:43.010664113 +0000 UTC" Oct 30 21:16:13.283812 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:13.283748 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12335h8m29.72692014s" Oct 30 21:16:14.879643 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:14.879077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:14.879643 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:14.879401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:14.879643 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:14.879286 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:14.879643 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:14.879509 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:14.879643 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:14.879531 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs podName:6df5ee35-2300-4a44-9ca3-c40545ab2866 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:18.87951108 +0000 UTC m=+9.336148444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs") pod "network-metrics-daemon-55bl6" (UID: "6df5ee35-2300-4a44-9ca3-c40545ab2866") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:14.879643 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:14.879576 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret podName:db28a65b-b841-4446-b062-9526c7e8139a nodeName:}" failed. No retries permitted until 2026-10-30 21:16:18.879556761 +0000 UTC m=+9.336194138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret") pod "global-pull-secret-syncer-bbwl8" (UID: "db28a65b-b841-4446-b062-9526c7e8139a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:14.980462 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:14.980435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:14.980625 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:14.980604 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:14.980689 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:14.980635 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:14.980689 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:14.980650 2574 projected.go:196] Error preparing data for projected volume kube-api-access-q7ccw for pod openshift-network-diagnostics/network-check-target-79bgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:14.980816 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:14.980714 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw podName:4401a9ff-fa39-4669-b5b1-3c9f473904cf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:18.980694863 +0000 UTC m=+9.437332238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7ccw" (UniqueName: "kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw") pod "network-check-target-79bgk" (UID: "4401a9ff-fa39-4669-b5b1-3c9f473904cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:15.081299 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:15.081265 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:15.081463 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:15.081437 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:15.081531 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:15.081483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:15.081610 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:15.081551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:15.081665 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:15.081628 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:15.081783 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:15.081748 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:15.108716 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:15.108686 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e8b937f14ab0703a8793557804f6cae" containerID="29e045af7c731199c539bf0dd225061d1358d0744e95210a7c3faa02cadd7b03" exitCode=0 Oct 30 21:16:15.108867 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:15.108752 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"2e8b937f14ab0703a8793557804f6cae","Type":"ContainerDied","Data":"29e045af7c731199c539bf0dd225061d1358d0744e95210a7c3faa02cadd7b03"} Oct 30 21:16:16.121096 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:16.121057 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"2e8b937f14ab0703a8793557804f6cae","Type":"ContainerStarted","Data":"afd83a8e0f80c9f479d27d629198ee7c69dce910d29999658a5b9c66ff2fbcf9"} Oct 30 21:16:16.139628 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:16.138839 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"a4207178a950303c256d4f333088d96f","Type":"ContainerStarted","Data":"147820bfa14f1419564ae6f75136e929a33ec5cf13e2d397b08896c729e7787f"} Oct 30 21:16:16.242901 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:16.242829 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" podStartSLOduration=5.242806996 podStartE2EDuration="5.242806996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:16.24158594 +0000 UTC m=+6.698223326" watchObservedRunningTime="2026-10-30 21:16:16.242806996 +0000 UTC m=+6.699444384" Oct 30 21:16:16.362954 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:16.362875 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podStartSLOduration=5.362852886 podStartE2EDuration="5.362852886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:16.362077494 +0000 UTC m=+6.818714879" watchObservedRunningTime="2026-10-30 21:16:16.362852886 +0000 UTC m=+6.819490272" Oct 30 21:16:17.080954 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:17.080909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:17.080954 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:17.080950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:17.081217 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:17.080909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:17.081217 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:17.081077 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:17.081217 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:17.081187 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:17.081365 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:17.081277 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:17.152383 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:17.152336 2574 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="c39863edac5c872a76e1b10714014e19ee0cde093637fd2a9cd1a451fa4fddaa" exitCode=0 Oct 30 21:16:17.153725 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:17.153695 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84tpc" event={"ID":"06b8bac2-6f8c-4a6a-9953-0573489e7846","Type":"ContainerDied","Data":"c39863edac5c872a76e1b10714014e19ee0cde093637fd2a9cd1a451fa4fddaa"} Oct 30 21:16:18.165717 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:18.165670 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tljvj" event={"ID":"62b9feff-59c5-45d5-a398-c78ad8dd590b","Type":"ContainerStarted","Data":"d88284a31b08651f1d7d037c4ae5831baebb6e1bc7df40fd3a663303b9a50b28"} Oct 30 21:16:18.289872 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:18.289812 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tljvj" podStartSLOduration=2.50504502 podStartE2EDuration="7.289788649s" totalImagesPullingTime="4.784743629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:18.28879316 +0000 UTC m=+8.745430548" watchObservedRunningTime="2026-10-30 21:16:18.289788649 +0000 UTC m=+8.746426037" Oct 30 21:16:18.917488 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:18.917400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:18.917700 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:18.917504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:18.917700 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:18.917674 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:18.917827 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:18.917753 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs podName:6df5ee35-2300-4a44-9ca3-c40545ab2866 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:26.917730174 +0000 UTC m=+17.374367535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs") pod "network-metrics-daemon-55bl6" (UID: "6df5ee35-2300-4a44-9ca3-c40545ab2866") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:18.917900 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:18.917860 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:18.917956 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:18.917906 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret podName:db28a65b-b841-4446-b062-9526c7e8139a nodeName:}" failed. No retries permitted until 2026-10-30 21:16:26.917893289 +0000 UTC m=+17.374530666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret") pod "global-pull-secret-syncer-bbwl8" (UID: "db28a65b-b841-4446-b062-9526c7e8139a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:19.018793 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:19.018383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:19.018793 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:19.018643 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:19.018793 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:19.018667 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:19.018793 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:19.018680 2574 projected.go:196] Error preparing data for projected volume kube-api-access-q7ccw for pod openshift-network-diagnostics/network-check-target-79bgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:19.018793 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:19.018746 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw podName:4401a9ff-fa39-4669-b5b1-3c9f473904cf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:27.01872484 +0000 UTC m=+17.475362225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7ccw" (UniqueName: "kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw") pod "network-check-target-79bgk" (UID: "4401a9ff-fa39-4669-b5b1-3c9f473904cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:19.081073 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:19.080491 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:19.081073 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:19.080666 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:19.081073 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:19.080764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:19.081073 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:19.080866 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:19.081073 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:19.080925 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:19.081073 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:19.081004 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:19.172219 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:19.172137 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xlrvq" event={"ID":"cb67e001-cef7-4f5e-9f38-2617953250d4","Type":"ContainerStarted","Data":"705b7d06a18b08e76868db946a11f4959029c269aa9eb44b970fbf9c290c9bad"} Oct 30 21:16:20.177506 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:20.177467 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" event={"ID":"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98","Type":"ContainerStarted","Data":"5f926a5cf755f7fce8e132b83535ed0726feeb11bda2f5b15648ef0bb417b400"} Oct 30 21:16:21.080426 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:21.080390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:21.080627 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:21.080551 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:21.080627 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:21.080576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:21.080775 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:21.080708 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:21.080775 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:21.080749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:21.080909 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:21.080857 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:21.671329 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:21.671267 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:16:21.807720 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:21.807690 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:21.808848 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:21.808823 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:21.934311 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:21.934248 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xlrvq" podStartSLOduration=4.965417182 podStartE2EDuration="10.93422739s" totalImagesPullingTime="5.968810208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:19.294883963 +0000 UTC m=+9.751521351" watchObservedRunningTime="2026-10-30 21:16:21.93422739 +0000 UTC m=+12.390864777" Oct 30 21:16:22.031010 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:22.030899 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:16:21.671297165Z","UUID":"048a30e3-c325-4d55-96ce-6c8a79978604","Handler":null,"Name":"","Endpoint":""} Oct 30 21:16:22.032732 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:22.032703 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:16:22.032867 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:22.032742 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:16:22.194232 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:22.194194 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kd826" event={"ID":"66b958bb-61d6-473c-bf4d-21fefe6e8675","Type":"ContainerStarted","Data":"4ea0c48a0387289bb09c5392ee40b86cafe000218116ccc61a8a135742c207db"} Oct 30 21:16:22.197438 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:22.196589 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpmnj" event={"ID":"82f30bd6-0ef2-47f0-8f55-6f5370a098e1","Type":"ContainerStarted","Data":"5d7d73229e5bf2972128e22ff1daa67b392cb0493da6bac87fc84fa7eaa260c9"} Oct 30 21:16:22.200545 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:22.200495 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" event={"ID":"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98","Type":"ContainerStarted","Data":"da48e045a5475d7fec3109b8feae93a205f17b8b99519efa6afbf1a3822239c4"} Oct 30 21:16:22.430387 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:22.430303 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zpmnj" podStartSLOduration=2.167979083 podStartE2EDuration="11.430289659s" totalImagesPullingTime="9.262310576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:22.430159268 +0000 UTC m=+12.886796650" watchObservedRunningTime="2026-10-30 21:16:22.430289659 +0000 UTC m=+12.886927040" Oct 30 21:16:22.430750 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:22.430720 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kd826" podStartSLOduration=2.713289896 podStartE2EDuration="11.4307112s" totalImagesPullingTime="8.717421304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:22.316150996 +0000 UTC m=+12.772788377" watchObservedRunningTime="2026-10-30 21:16:22.4307112 +0000 UTC m=+12.887348599" Oct 30 21:16:23.080672 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:23.080588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:23.081133 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:23.080729 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:23.081133 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:23.080833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:23.081133 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:23.080910 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:23.081133 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:23.080959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:23.081133 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:23.081008 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:23.205097 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:23.205058 2574 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="429977a5adbd4d1d8a953a7bbf757f86acdc1183544cd5fd250eca4d98977567" exitCode=0 Oct 30 21:16:23.205241 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:23.205129 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84tpc" event={"ID":"06b8bac2-6f8c-4a6a-9953-0573489e7846","Type":"ContainerDied","Data":"429977a5adbd4d1d8a953a7bbf757f86acdc1183544cd5fd250eca4d98977567"} Oct 30 21:16:23.209053 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:23.208614 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6r5rg" event={"ID":"31857e99-9c08-45ea-85da-f154ca58223d","Type":"ContainerStarted","Data":"c18687a3b5d44d5c53f35bfb8ce563ccf3b2111997b4486e185c5068f7bc4deb"} Oct 30 21:16:23.436785 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:23.436470 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6r5rg" podStartSLOduration=3.710707362 podStartE2EDuration="12.436453629s" totalImagesPullingTime="8.725746267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:23.436441864 +0000 UTC m=+13.893079244" watchObservedRunningTime="2026-10-30 21:16:23.436453629 +0000 UTC m=+13.893091009" Oct 30 21:16:24.212688 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:24.212651 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" event={"ID":"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98","Type":"ContainerStarted","Data":"a71805ba9e0f7c54a28e4d4d43aaac02d120f2884300f5092cd3f2a9f1551e87"} Oct 30 21:16:24.214180 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:24.214154 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" event={"ID":"7b8cf76d-e722-49b5-8c62-fe88eff85691","Type":"ContainerStarted","Data":"69d497c1a542b126314461414a863fd04050fc1853016c3250f569407d317d86"} Oct 30 21:16:24.336730 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:24.336684 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" podStartSLOduration=3.075925419 podStartE2EDuration="13.336670548s" totalImagesPullingTime="10.260745129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:24.336206341 +0000 UTC m=+14.792843723" watchObservedRunningTime="2026-10-30 21:16:24.336670548 +0000 UTC m=+14.793307930" Oct 30 21:16:24.452761 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:24.452697 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" podStartSLOduration=2.252092829 podStartE2EDuration="13.452681844s" totalImagesPullingTime="11.200589015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:24.451898413 +0000 UTC m=+14.908535794" watchObservedRunningTime="2026-10-30 21:16:24.452681844 +0000 UTC m=+14.909319225" Oct 30 21:16:25.081133 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:25.081099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:25.081133 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:25.081125 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:25.081334 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:25.081150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:25.081334 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:25.081230 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:25.081334 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:25.081295 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:25.081467 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:25.081386 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:25.218667 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:25.218632 2574 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="c1a18d7c2eb01268fb333d4f1d0b793a72724625eefe655a7cfa4f85a4c4947d" exitCode=0 Oct 30 21:16:25.219202 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:25.218712 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84tpc" event={"ID":"06b8bac2-6f8c-4a6a-9953-0573489e7846","Type":"ContainerDied","Data":"c1a18d7c2eb01268fb333d4f1d0b793a72724625eefe655a7cfa4f85a4c4947d"} Oct 30 21:16:26.986773 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:26.986738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:26.987384 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:26.986800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:26.987384 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:26.986894 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:26.987384 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:26.986898 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:26.987384 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:26.986958 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret podName:db28a65b-b841-4446-b062-9526c7e8139a nodeName:}" failed. No retries permitted until 2026-10-30 21:16:42.986940985 +0000 UTC m=+33.443578349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret") pod "global-pull-secret-syncer-bbwl8" (UID: "db28a65b-b841-4446-b062-9526c7e8139a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:26.987384 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:26.986978 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs podName:6df5ee35-2300-4a44-9ca3-c40545ab2866 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:42.986968291 +0000 UTC m=+33.443605652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs") pod "network-metrics-daemon-55bl6" (UID: "6df5ee35-2300-4a44-9ca3-c40545ab2866") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:27.080570 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:27.080540 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:27.080570 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:27.080567 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:27.080779 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:27.080670 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:27.080779 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:27.080727 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:27.080901 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:27.080780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:27.080901 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:27.080844 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:27.087669 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:27.087644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:27.087826 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:27.087794 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:27.087826 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:27.087808 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:27.087826 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:27.087818 2574 projected.go:196] Error preparing data for projected volume kube-api-access-q7ccw for pod openshift-network-diagnostics/network-check-target-79bgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:27.087993 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:27.087868 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw podName:4401a9ff-fa39-4669-b5b1-3c9f473904cf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:43.087854936 +0000 UTC m=+33.544492295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7ccw" (UniqueName: "kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw") pod "network-check-target-79bgk" (UID: "4401a9ff-fa39-4669-b5b1-3c9f473904cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:27.225927 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:27.225814 2574 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="af1fc20218c6ff619d1f8f91a1342e1256d4f58df2395f9c4bf98a485af19d74" exitCode=0 Oct 30 21:16:27.225927 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:27.225876 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84tpc" event={"ID":"06b8bac2-6f8c-4a6a-9953-0573489e7846","Type":"ContainerDied","Data":"af1fc20218c6ff619d1f8f91a1342e1256d4f58df2395f9c4bf98a485af19d74"} Oct 30 21:16:28.232618 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.232370 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"fd5b7e79980a1acd8e6e51f1261517a1de40c0cc3fd71e0a79ba31021d124f32"} Oct 30 21:16:28.232618 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.232612 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"a1834db24d17f656e4dd9e0332b5546b7a2e3a82be07c963622499eb3cb8944a"} Oct 30 21:16:28.233141 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.232645 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"59a288a956ab619ae337156b7384da918808b325ce5f2264eebcefdff835592f"} Oct 30 21:16:28.233141 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.232673 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"48a32113bef25c9970f947449b6ff0dfc4d15f059e0205c6411098a318de9f79"} Oct 30 21:16:28.233141 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.232712 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"d37c93e4e2ed7bc8d954fc84a3829169dc47d1a4b53dfdc3d7e26ee0bd8efcf0"} Oct 30 21:16:28.233141 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.232757 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"836d8320e2109129aa2fc00dd31987fc2a287ff1c82f6b39327f0d7fb1748123"} Oct 30 21:16:28.618449 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.618367 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:28.618597 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.618535 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:16:28.618946 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:28.618930 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:29.080789 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:29.080704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:29.080789 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:29.080728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:29.080789 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:29.080737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:29.081042 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:29.080840 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:29.081042 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:29.080904 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:29.081042 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:29.080996 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:30.239996 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:30.239829 2574 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="87e6e971180955af2746255e903a357e3a99f67d34c5d4c202901d5f011be932" exitCode=0 Oct 30 21:16:30.240363 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:30.239906 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84tpc" event={"ID":"06b8bac2-6f8c-4a6a-9953-0573489e7846","Type":"ContainerDied","Data":"87e6e971180955af2746255e903a357e3a99f67d34c5d4c202901d5f011be932"} Oct 30 21:16:30.246596 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:30.246551 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"0eb9abf4ffa1c7e504089f3f824c16331f671420fd99ce19c6935e0e55159598"} Oct 30 21:16:31.080380 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:31.080348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:31.080380 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:31.080367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:31.080595 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:31.080360 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:31.080595 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:31.080455 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:31.080595 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:31.080534 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:31.080679 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:31.080619 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:31.251458 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:31.251422 2574 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="5021aa801c4085afd539fb1b26ab584b12ca18e868b8eb57d0df3433d517860f" exitCode=0 Oct 30 21:16:31.251918 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:31.251472 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84tpc" event={"ID":"06b8bac2-6f8c-4a6a-9953-0573489e7846","Type":"ContainerDied","Data":"5021aa801c4085afd539fb1b26ab584b12ca18e868b8eb57d0df3433d517860f"} Oct 30 21:16:32.257074 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:32.257040 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-84tpc" event={"ID":"06b8bac2-6f8c-4a6a-9953-0573489e7846","Type":"ContainerStarted","Data":"f9947bee49526853cb6e20a559e71fcae46fbb20ac2b02bd26d8fcdc1afd94f3"} Oct 30 21:16:32.476344 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:32.476286 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-84tpc" podStartSLOduration=8.508841666 podStartE2EDuration="21.476267969s" totalImagesPullingTime="12.967426303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:32.475648605 +0000 UTC m=+22.932285987" watchObservedRunningTime="2026-10-30 21:16:32.476267969 +0000 UTC m=+22.932905350" Oct 30 21:16:33.080378 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:33.080339 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:33.080554 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:33.080399 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:33.080554 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:33.080406 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:33.080554 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:33.080473 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:33.080681 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:33.080592 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:33.080681 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:33.080654 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:33.262665 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:33.262622 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" event={"ID":"2e226384-4104-4da9-a6b9-ce21038fedc3","Type":"ContainerStarted","Data":"6629c160d31eef4a193c5c8ef59ad94d4c56803690ed04091715ae445322aea8"} Oct 30 21:16:33.533331 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:33.533232 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" podStartSLOduration=7.200992082 podStartE2EDuration="22.533218246s" totalImagesPullingTime="15.332226164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:33.533097903 +0000 UTC m=+23.989735285" watchObservedRunningTime="2026-10-30 21:16:33.533218246 +0000 UTC m=+23.989855628" Oct 30 21:16:34.265634 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:34.265605 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:34.266092 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:34.265692 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:34.266092 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:34.265742 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:34.281820 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:34.281785 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:34.281962 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:34.281916 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:35.080799 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.080538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:35.080799 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.080551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:35.081002 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:35.080831 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:35.081002 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.080598 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:35.081002 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:35.080884 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:35.081002 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:35.080957 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:35.201677 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.201640 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bbwl8"] Oct 30 21:16:35.205011 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.204931 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-55bl6"] Oct 30 21:16:35.205950 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.205927 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-79bgk"] Oct 30 21:16:35.267569 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.267539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:35.268022 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:35.267637 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:35.268022 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.267672 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:35.268022 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:35.267542 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:35.268022 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:35.267810 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:35.268022 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:35.267864 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:37.080547 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:37.080505 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:37.081039 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:37.080568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:37.081039 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:37.080521 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:37.081039 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:37.080672 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:37.081039 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:37.080735 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:37.081039 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:37.080800 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:39.081083 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:39.081050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:39.081571 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:39.081097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:39.081571 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:39.081097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:39.081571 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:39.081210 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbwl8" podUID="db28a65b-b841-4446-b062-9526c7e8139a" Oct 30 21:16:39.081571 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:39.081228 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79bgk" podUID="4401a9ff-fa39-4669-b5b1-3c9f473904cf" Oct 30 21:16:39.081571 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:39.081324 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-55bl6" podUID="6df5ee35-2300-4a44-9ca3-c40545ab2866" Oct 30 21:16:40.320324 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.320247 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeReady" Oct 30 21:16:40.320718 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.320379 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:16:40.474609 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.474576 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qc6qf"] Oct 30 21:16:40.476243 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.476228 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-chx8w"] Oct 30 21:16:40.476362 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.476344 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:40.477828 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.477808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.494821 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.494798 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qc6qf"] Oct 30 21:16:40.497457 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.497435 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-chx8w"] Oct 30 21:16:40.529637 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.529600 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:16:40.529788 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.529707 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:16:40.529788 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.529719 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:16:40.529788 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.529712 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:16:40.529788 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.529726 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wvjk\"" Oct 30 21:16:40.529788 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.529759 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:16:40.529788 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.529790 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42q2c\"" Oct 30 21:16:40.580270 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.580213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2923a9-10b7-4e7a-a828-3b8145945ba6-cert\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:40.580270 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.580245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62942b-039a-4dd9-8fe2-609094d6822c-tmp-dir\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.580396 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.580280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62942b-039a-4dd9-8fe2-609094d6822c-config-volume\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.580396 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.580355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzgj\" (UniqueName: \"kubernetes.io/projected/eb62942b-039a-4dd9-8fe2-609094d6822c-kube-api-access-cmzgj\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.580470 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.580439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6sr4\" (UniqueName: \"kubernetes.io/projected/6f2923a9-10b7-4e7a-a828-3b8145945ba6-kube-api-access-p6sr4\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:40.580528 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.580478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.680852 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.680825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6sr4\" (UniqueName: \"kubernetes.io/projected/6f2923a9-10b7-4e7a-a828-3b8145945ba6-kube-api-access-p6sr4\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:40.681037 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.680872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.681037 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.680906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2923a9-10b7-4e7a-a828-3b8145945ba6-cert\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:40.681037 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.680925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62942b-039a-4dd9-8fe2-609094d6822c-tmp-dir\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.681037 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.680953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62942b-039a-4dd9-8fe2-609094d6822c-config-volume\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.681037 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.680977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzgj\" (UniqueName: \"kubernetes.io/projected/eb62942b-039a-4dd9-8fe2-609094d6822c-kube-api-access-cmzgj\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.681276 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:40.681061 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:16:40.681276 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:40.681146 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls podName:eb62942b-039a-4dd9-8fe2-609094d6822c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:41.181123751 +0000 UTC m=+31.637761120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls") pod "dns-default-chx8w" (UID: "eb62942b-039a-4dd9-8fe2-609094d6822c") : secret "dns-default-metrics-tls" not found Oct 30 21:16:40.681368 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.681320 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62942b-039a-4dd9-8fe2-609094d6822c-tmp-dir\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.681629 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.681611 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62942b-039a-4dd9-8fe2-609094d6822c-config-volume\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.685477 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.685356 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2923a9-10b7-4e7a-a828-3b8145945ba6-cert\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:40.738779 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.738743 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzgj\" (UniqueName: \"kubernetes.io/projected/eb62942b-039a-4dd9-8fe2-609094d6822c-kube-api-access-cmzgj\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:40.739372 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.739352 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6sr4\" (UniqueName: \"kubernetes.io/projected/6f2923a9-10b7-4e7a-a828-3b8145945ba6-kube-api-access-p6sr4\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:40.787449 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:40.787417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:41.081236 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.081183 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:41.081236 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.081240 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:41.081446 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.081202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:41.134064 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.134025 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:16:41.134064 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.134034 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:16:41.134354 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.134122 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:16:41.134354 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.134214 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5jmhs\"" Oct 30 21:16:41.134354 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.134229 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p2m87\"" Oct 30 21:16:41.134354 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.134279 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:16:41.134354 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.134280 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lps2\"" Oct 30 21:16:41.166684 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.166654 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qc6qf"] Oct 30 21:16:41.170584 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:41.170559 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2923a9_10b7_4e7a_a828_3b8145945ba6.slice/crio-6fa9cc923e9b48a0c990cffbebd9364dad580c0a1f1b36506d01beec22114354 WatchSource:0}: Error finding container 6fa9cc923e9b48a0c990cffbebd9364dad580c0a1f1b36506d01beec22114354: Status 404 returned error can't find the container with id 6fa9cc923e9b48a0c990cffbebd9364dad580c0a1f1b36506d01beec22114354 Oct 30 21:16:41.184500 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.184454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:41.184634 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:41.184621 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:16:41.184686 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:41.184677 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls podName:eb62942b-039a-4dd9-8fe2-609094d6822c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:42.184665492 +0000 UTC m=+32.641302850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls") pod "dns-default-chx8w" (UID: "eb62942b-039a-4dd9-8fe2-609094d6822c") : secret "dns-default-metrics-tls" not found Oct 30 21:16:41.282689 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:41.282652 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qc6qf" event={"ID":"6f2923a9-10b7-4e7a-a828-3b8145945ba6","Type":"ContainerStarted","Data":"6fa9cc923e9b48a0c990cffbebd9364dad580c0a1f1b36506d01beec22114354"} Oct 30 21:16:42.191674 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.191634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:42.192320 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:42.191780 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:16:42.192320 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:42.191835 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls podName:eb62942b-039a-4dd9-8fe2-609094d6822c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:44.191822378 +0000 UTC m=+34.648459737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls") pod "dns-default-chx8w" (UID: "eb62942b-039a-4dd9-8fe2-609094d6822c") : secret "dns-default-metrics-tls" not found Oct 30 21:16:42.410794 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.410760 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4z7zj"] Oct 30 21:16:42.412811 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.412789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.465106 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.465030 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:16:42.465318 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.465299 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:16:42.465393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.465304 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:16:42.465393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.465354 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:16:42.465393 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.465360 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7dj2\"" Oct 30 21:16:42.473132 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.473106 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4z7zj"] Oct 30 21:16:42.494381 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.494343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823598bc-abd8-49f6-91d9-2eb21ec1435c-crio-socket\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.494581 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.494389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92wb2\" (UniqueName: \"kubernetes.io/projected/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-api-access-92wb2\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.494581 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.494564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.494694 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.494617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/823598bc-abd8-49f6-91d9-2eb21ec1435c-data-volume\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.494694 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.494637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.595939 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.595897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.596125 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.595983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/823598bc-abd8-49f6-91d9-2eb21ec1435c-data-volume\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.596125 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.596011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.596125 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:42.596024 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:16:42.596125 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:42.596105 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:43.096084873 +0000 UTC m=+33.552722251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : configmap "kube-rbac-proxy" not found Oct 30 21:16:42.596328 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:42.596152 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:16:42.596328 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.596174 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823598bc-abd8-49f6-91d9-2eb21ec1435c-crio-socket\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.596328 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:42.596216 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:43.096199947 +0000 UTC m=+33.552837321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:16:42.596328 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.596235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92wb2\" (UniqueName: \"kubernetes.io/projected/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-api-access-92wb2\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.596531 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.596407 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823598bc-abd8-49f6-91d9-2eb21ec1435c-crio-socket\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.596531 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.596453 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/823598bc-abd8-49f6-91d9-2eb21ec1435c-data-volume\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.661642 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.661610 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92wb2\" (UniqueName: \"kubernetes.io/projected/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-api-access-92wb2\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:42.662103 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.662084 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-cpc25-sxdlt"] Oct 30 21:16:42.664006 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.663987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.721004 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.720904 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k4xtg\"" Oct 30 21:16:42.722082 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.722055 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cpc25-sxdlt"] Oct 30 21:16:42.722578 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.722561 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:16:42.773398 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.773366 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:16:42.795614 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.795583 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z"] Oct 30 21:16:42.797359 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.797330 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" Oct 30 21:16:42.797547 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.797523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.797634 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.797589 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.797695 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.797634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.797695 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.797661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.850046 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.850011 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pc6p4\"" Oct 30 21:16:42.857360 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.856181 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:16:42.857360 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.856564 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:16:42.857360 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.857309 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z"] Oct 30 21:16:42.898699 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.898665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.898862 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.898735 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq78x\" (UniqueName: \"kubernetes.io/projected/9b3f89fa-e28d-4591-9fe3-0a1e06b61984-kube-api-access-rq78x\") pod \"migrator-66df5d496b-7tz9z\" (UID: \"9b3f89fa-e28d-4591-9fe3-0a1e06b61984\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" Oct 30 21:16:42.898862 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.898765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.898862 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.898820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.898862 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.898851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.899194 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.899146 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.899609 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.899590 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.899686 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.899613 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.956342 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.956315 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.975738 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.975667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:42.999912 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.999882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:43.000035 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.999957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:43.000035 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:42.999983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq78x\" (UniqueName: \"kubernetes.io/projected/9b3f89fa-e28d-4591-9fe3-0a1e06b61984-kube-api-access-rq78x\") pod \"migrator-66df5d496b-7tz9z\" (UID: \"9b3f89fa-e28d-4591-9fe3-0a1e06b61984\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" Oct 30 21:16:43.000384 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:43.000365 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:16:43.000511 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:43.000428 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs podName:6df5ee35-2300-4a44-9ca3-c40545ab2866 nodeName:}" failed. No retries permitted until 2026-10-30 21:17:15.00040919 +0000 UTC m=+65.457046583 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs") pod "network-metrics-daemon-55bl6" (UID: "6df5ee35-2300-4a44-9ca3-c40545ab2866") : secret "metrics-daemon-secret" not found Oct 30 21:16:43.003358 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.003335 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:43.061469 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.061440 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq78x\" (UniqueName: \"kubernetes.io/projected/9b3f89fa-e28d-4591-9fe3-0a1e06b61984-kube-api-access-rq78x\") pod \"migrator-66df5d496b-7tz9z\" (UID: \"9b3f89fa-e28d-4591-9fe3-0a1e06b61984\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" Oct 30 21:16:43.100437 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.100397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:43.100657 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.100451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:43.100657 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.100482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:43.100657 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:43.100560 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:16:43.100657 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:43.100632 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:16:43.100657 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:43.100641 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:44.100621738 +0000 UTC m=+34.557259120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : configmap "kube-rbac-proxy" not found Oct 30 21:16:43.100903 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:43.100677 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:44.100666042 +0000 UTC m=+34.557303400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:16:43.103482 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.103455 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:43.109142 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.109114 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" Oct 30 21:16:43.111452 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.111428 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sg4nr"] Oct 30 21:16:43.113465 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.113439 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.167682 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.167658 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:16:43.167789 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.167750 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:16:43.167856 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.167812 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ptpxj\"" Oct 30 21:16:43.167856 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.167850 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:16:43.167959 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.167904 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:16:43.174511 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.173161 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sg4nr"] Oct 30 21:16:43.197719 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.197683 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:43.203992 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.203956 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:43.205715 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.205687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxbt\" (UniqueName: \"kubernetes.io/projected/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-kube-api-access-hsxbt\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.205848 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.205741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-key\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.206071 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.206053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-cabundle\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.306912 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.306879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-cabundle\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.307073 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.306932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxbt\" (UniqueName: \"kubernetes.io/projected/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-kube-api-access-hsxbt\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.307073 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.306967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-key\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.307701 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.307674 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-cabundle\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.309845 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.309815 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-key\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.355169 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.355138 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cpc25-sxdlt"] Oct 30 21:16:43.358553 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:43.358523 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8023590c_1d50_48e6_93e4_683b8368c13c.slice/crio-cc60dfa3b6d1921fa964050e1fdde4e3377d80e99643e5f43eed09ae6a6e4b5c WatchSource:0}: Error finding container cc60dfa3b6d1921fa964050e1fdde4e3377d80e99643e5f43eed09ae6a6e4b5c: Status 404 returned error can't find the container with id cc60dfa3b6d1921fa964050e1fdde4e3377d80e99643e5f43eed09ae6a6e4b5c Oct 30 21:16:43.364115 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.364070 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxbt\" (UniqueName: \"kubernetes.io/projected/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-kube-api-access-hsxbt\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.427006 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.426980 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:43.483614 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.483531 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z"] Oct 30 21:16:43.490757 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:43.490723 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3f89fa_e28d_4591_9fe3_0a1e06b61984.slice/crio-0abca98bbf95cafa478447fdfdbb0fd3faac498464e081dea3409bca2d1e0294 WatchSource:0}: Error finding container 0abca98bbf95cafa478447fdfdbb0fd3faac498464e081dea3409bca2d1e0294: Status 404 returned error can't find the container with id 0abca98bbf95cafa478447fdfdbb0fd3faac498464e081dea3409bca2d1e0294 Oct 30 21:16:43.578862 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.578782 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bbwl8"] Oct 30 21:16:43.582552 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.582527 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-79bgk"] Oct 30 21:16:43.582552 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:43.582547 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb28a65b_b841_4446_b062_9526c7e8139a.slice/crio-ce483c49732a81f80cbe3b1781477093ce28e60126d33ebd351d3ba0656cb445 WatchSource:0}: Error finding container ce483c49732a81f80cbe3b1781477093ce28e60126d33ebd351d3ba0656cb445: Status 404 returned error can't find the container with id ce483c49732a81f80cbe3b1781477093ce28e60126d33ebd351d3ba0656cb445 Oct 30 21:16:43.594088 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:43.594057 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4401a9ff_fa39_4669_b5b1_3c9f473904cf.slice/crio-66f94439360da0e59db67eed651b2ab1a5e735c801459b980e2f82a48dbf8578 WatchSource:0}: Error finding container 66f94439360da0e59db67eed651b2ab1a5e735c801459b980e2f82a48dbf8578: Status 404 returned error can't find the container with id 66f94439360da0e59db67eed651b2ab1a5e735c801459b980e2f82a48dbf8578 Oct 30 21:16:43.802113 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:43.802079 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sg4nr"] Oct 30 21:16:43.805100 ip-10-0-128-26 kubenswrapper[2574]: W1030 21:16:43.805072 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d4bf6f_7b7e_45a5_a866_87c76eb414ab.slice/crio-fd01fda8ff3b47b0d7ffd7cb3f9a7f3bc1f9c8ea5a4b479d62b894b73b9ab631 WatchSource:0}: Error finding container fd01fda8ff3b47b0d7ffd7cb3f9a7f3bc1f9c8ea5a4b479d62b894b73b9ab631: Status 404 returned error can't find the container with id fd01fda8ff3b47b0d7ffd7cb3f9a7f3bc1f9c8ea5a4b479d62b894b73b9ab631 Oct 30 21:16:44.113344 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.113306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:44.113519 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.113354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:44.113519 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:44.113435 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:16:44.113519 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:44.113510 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:16:44.113633 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:44.113525 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:46.113502824 +0000 UTC m=+36.570140197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : configmap "kube-rbac-proxy" not found Oct 30 21:16:44.113633 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:44.113551 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:46.113542786 +0000 UTC m=+36.570180145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:16:44.214774 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.214736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:44.215212 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:44.214918 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:16:44.215212 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:44.215000 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls podName:eb62942b-039a-4dd9-8fe2-609094d6822c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:48.214978786 +0000 UTC m=+38.671616146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls") pod "dns-default-chx8w" (UID: "eb62942b-039a-4dd9-8fe2-609094d6822c") : secret "dns-default-metrics-tls" not found Oct 30 21:16:44.293224 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.293142 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qc6qf" event={"ID":"6f2923a9-10b7-4e7a-a828-3b8145945ba6","Type":"ContainerStarted","Data":"f885486229ca855a871dcaf5bc4346d1c451c85e4eb1f3ab136d9026a3e88e87"} Oct 30 21:16:44.294779 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.294744 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" event={"ID":"8023590c-1d50-48e6-93e4-683b8368c13c","Type":"ContainerStarted","Data":"cc60dfa3b6d1921fa964050e1fdde4e3377d80e99643e5f43eed09ae6a6e4b5c"} Oct 30 21:16:44.296089 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.296054 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" event={"ID":"21d4bf6f-7b7e-45a5-a866-87c76eb414ab","Type":"ContainerStarted","Data":"fd01fda8ff3b47b0d7ffd7cb3f9a7f3bc1f9c8ea5a4b479d62b894b73b9ab631"} Oct 30 21:16:44.297349 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.297293 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-79bgk" event={"ID":"4401a9ff-fa39-4669-b5b1-3c9f473904cf","Type":"ContainerStarted","Data":"66f94439360da0e59db67eed651b2ab1a5e735c801459b980e2f82a48dbf8578"} Oct 30 21:16:44.298739 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.298716 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bbwl8" event={"ID":"db28a65b-b841-4446-b062-9526c7e8139a","Type":"ContainerStarted","Data":"ce483c49732a81f80cbe3b1781477093ce28e60126d33ebd351d3ba0656cb445"} Oct 30 21:16:44.299886 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.299845 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" event={"ID":"9b3f89fa-e28d-4591-9fe3-0a1e06b61984","Type":"ContainerStarted","Data":"0abca98bbf95cafa478447fdfdbb0fd3faac498464e081dea3409bca2d1e0294"} Oct 30 21:16:44.409949 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:44.409888 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qc6qf" podStartSLOduration=1.5544033750000001 podStartE2EDuration="4.409868713s" totalImagesPullingTime="2.855465338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:44.409280036 +0000 UTC m=+34.865917416" watchObservedRunningTime="2026-10-30 21:16:44.409868713 +0000 UTC m=+34.866506094" Oct 30 21:16:46.130647 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:46.130586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:46.131128 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:46.130682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:46.131128 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:46.130817 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:16:46.131128 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:46.130874 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:50.130859735 +0000 UTC m=+40.587497106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:16:46.131305 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:46.131172 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:16:46.131305 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:46.131220 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:50.131203476 +0000 UTC m=+40.587840839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : configmap "kube-rbac-proxy" not found Oct 30 21:16:46.308085 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:46.308043 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" event={"ID":"9b3f89fa-e28d-4591-9fe3-0a1e06b61984","Type":"ContainerStarted","Data":"ce8acbbd0fdbeded88e1dbeeb1f1add7193f1c57c9ddc4c09b8c2745b00acf79"} Oct 30 21:16:47.314222 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:47.314117 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" event={"ID":"8023590c-1d50-48e6-93e4-683b8368c13c","Type":"ContainerStarted","Data":"285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771"} Oct 30 21:16:47.429134 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:47.429080 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" podStartSLOduration=3.490071394 podStartE2EDuration="5.429062014s" totalImagesPullingTime="1.93899062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:46.422987676 +0000 UTC m=+36.879625057" watchObservedRunningTime="2026-10-30 21:16:47.429062014 +0000 UTC m=+37.885699396" Oct 30 21:16:47.429433 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:47.429402 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" podStartSLOduration=2.390362965 podStartE2EDuration="5.429390695s" totalImagesPullingTime="3.03902773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:47.429386477 +0000 UTC m=+37.886023861" watchObservedRunningTime="2026-10-30 21:16:47.429390695 +0000 UTC m=+37.886028078" Oct 30 21:16:48.253068 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:48.252886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:48.253230 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:48.253079 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:16:48.253230 ip-10-0-128-26 kubenswrapper[2574]: E1030 21:16:48.253163 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls podName:eb62942b-039a-4dd9-8fe2-609094d6822c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:56.253141496 +0000 UTC m=+46.709778893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls") pod "dns-default-chx8w" (UID: "eb62942b-039a-4dd9-8fe2-609094d6822c") : secret "dns-default-metrics-tls" not found Oct 30 21:16:48.318137 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:48.317680 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" event={"ID":"21d4bf6f-7b7e-45a5-a866-87c76eb414ab","Type":"ContainerStarted","Data":"523a2c2cd8209246094afec476257b2067c026ee631a1003046ed9a86501c560"} Oct 30 21:16:48.319422 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:48.319393 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-79bgk" event={"ID":"4401a9ff-fa39-4669-b5b1-3c9f473904cf","Type":"ContainerStarted","Data":"376e52229d27138062261fe2c38c013d12835e02faadffcc354b332bfd7bab82"} Oct 30 21:16:48.432895 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:48.432144 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" podStartSLOduration=1.859278968 podStartE2EDuration="5.432126046s" totalImagesPullingTime="3.572847078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:48.431206618 +0000 UTC m=+38.887844000" watchObservedRunningTime="2026-10-30 21:16:48.432126046 +0000 UTC m=+38.888763428" Oct 30 21:16:48.544083 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:48.543979 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-79bgk" podStartSLOduration=33.215068026 podStartE2EDuration="37.54396293s" totalImagesPullingTime="4.328894904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:48.543676943 +0000 UTC m=+39.000314325" watchObservedRunningTime="2026-10-30 21:16:48.54396293 +0000 UTC m=+39.000600312" Oct 30 21:16:49.323323 ip-10-0-128-26 kubenswrapper[2574]: I1030 21:16:49.323290 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:49.802400 ip-10-0-128-26 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:16:49.823897 ip-10-0-128-26 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:16:49.824056 ip-10-0-128-26 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:16:49.824107 ip-10-0-128-26 systemd[1]: kubelet.service: Consumed 2.485s CPU time. Oct 30 21:16:49.834161 ip-10-0-128-26 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:16:49.896365 ip-10-0-128-26 kubenswrapper[4526]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:49.896365 ip-10-0-128-26 kubenswrapper[4526]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:16:49.896365 ip-10-0-128-26 kubenswrapper[4526]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:49.896365 ip-10-0-128-26 kubenswrapper[4526]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:49.898540 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898436 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:49.898540 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898539 4526 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:49.898540 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898543 4526 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898547 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898550 4526 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898552 4526 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898555 4526 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898558 4526 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898561 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898564 4526 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898566 4526 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898569 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898572 4526 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898574 4526 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898583 4526 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898586 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898589 4526 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898591 4526 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898594 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898596 4526 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898599 4526 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:49.898640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898602 4526 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898604 4526 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898607 4526 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898609 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898612 4526 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898615 4526 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898618 4526 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898621 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898624 4526 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898627 4526 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898630 4526 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898633 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898635 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898638 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898641 4526 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898643 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898647 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898649 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898652 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898655 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:49.899112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898657 4526 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898661 4526 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898666 4526 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898669 4526 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898672 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898675 4526 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898679 4526 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898682 4526 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898685 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898688 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898690 4526 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898693 4526 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898696 4526 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898698 4526 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898701 4526 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898703 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898706 4526 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898709 4526 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898712 4526 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:49.899632 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898715 4526 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898718 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898720 4526 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898723 4526 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898726 4526 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898729 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898731 4526 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898736 4526 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898739 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898742 4526 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898746 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898749 4526 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898752 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898754 4526 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898757 4526 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898759 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898762 4526 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898765 4526 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898767 4526 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:49.900239 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898770 4526 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898779 4526 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898782 4526 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898784 4526 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898787 4526 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898790 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898792 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898795 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898798 4526 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898800 4526 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898803 4526 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898806 4526 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898809 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898812 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898815 4526 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898817 4526 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898820 4526 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898823 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898826 4526 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898829 4526 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:49.900734 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898832 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898835 4526 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898837 4526 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898840 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898842 4526 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898845 4526 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.898847 4526 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899655 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899665 4526 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899668 4526 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899672 4526 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899674 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899677 4526 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899680 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899683 4526 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899686 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899689 4526 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899692 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899695 4526 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899697 4526 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:49.901229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899700 4526 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899703 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899706 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899708 4526 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899711 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899713 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899716 4526 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899719 4526 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899721 4526 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899724 4526 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899726 4526 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899729 4526 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899731 4526 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899734 4526 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899736 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899739 4526 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899742 4526 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899744 4526 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899747 4526 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899750 4526 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:49.901784 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899753 4526 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899755 4526 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899758 4526 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899760 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899763 4526 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899765 4526 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899768 4526 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899771 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899774 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899777 4526 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899780 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899782 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899785 4526 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899787 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899790 4526 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899792 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899795 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899797 4526 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899800 4526 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899802 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:49.902279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899805 4526 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899807 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899810 4526 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899813 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899815 4526 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899818 4526 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899821 4526 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899823 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899826 4526 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899829 4526 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899832 4526 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899834 4526 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899837 4526 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899839 4526 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899844 4526 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899847 4526 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899851 4526 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899853 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899856 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:49.902797 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899859 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899862 4526 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899865 4526 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899868 4526 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899870 4526 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899873 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899875 4526 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899878 4526 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899880 4526 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899883 4526 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899886 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899888 4526 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899891 4526 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899893 4526 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899896 4526 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899898 4526 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899902 4526 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899906 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899911 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:49.903265 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899914 4526 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899917 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899920 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899923 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899925 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899928 4526 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899930 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899933 4526 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899936 4526 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899938 4526 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899940 4526 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899943 4526 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899945 4526 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899948 4526 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.899950 4526 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900427 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900432 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900435 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900438 4526 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:49.903781 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900440 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900443 4526 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900446 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900448 4526 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900451 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900453 4526 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900456 4526 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900459 4526 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900461 4526 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900464 4526 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900467 4526 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900469 4526 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900472 4526 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900474 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900477 4526 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900502 4526 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900507 4526 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900511 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900515 4526 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900519 4526 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:49.904252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900522 4526 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900524 4526 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900527 4526 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900529 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900532 4526 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900535 4526 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900537 4526 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900540 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900543 4526 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900547 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900550 4526 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900552 4526 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900555 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900558 4526 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900560 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900573 4526 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900576 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900579 4526 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900582 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:49.904767 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900584 4526 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900588 4526 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900592 4526 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900595 4526 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900597 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900600 4526 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900602 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900605 4526 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900609 4526 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900612 4526 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900615 4526 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900617 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900620 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900623 4526 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900625 4526 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900628 4526 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900630 4526 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900633 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900635 4526 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900638 4526 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:49.905304 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900641 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900644 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900647 4526 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900650 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900653 4526 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900656 4526 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900658 4526 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900661 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900663 4526 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900666 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900668 4526 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900671 4526 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900673 4526 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900676 4526 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900679 4526 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900681 4526 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900684 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900686 4526 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900689 4526 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900691 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:49.906200 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900695 4526 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900698 4526 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900703 4526 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900707 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900710 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900712 4526 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900715 4526 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900718 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900720 4526 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900723 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900725 4526 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900728 4526 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900731 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900733 4526 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900736 4526 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900740 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900742 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900745 4526 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900748 4526 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:49.906812 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900752 4526 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900754 4526 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900757 4526 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.900759 4526 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900853 4526 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900862 4526 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900871 4526 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900878 4526 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900884 4526 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900889 4526 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900893 4526 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900897 4526 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900900 4526 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900905 4526 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900908 4526 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900912 4526 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900915 4526 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900918 4526 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900921 4526 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900925 4526 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900928 4526 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900931 4526 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900935 4526 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900938 4526 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:16:49.907307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900941 4526 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900945 4526 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900948 4526 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900952 4526 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900955 4526 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900959 4526 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900962 4526 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900965 4526 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900968 4526 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900971 4526 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900974 4526 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900977 4526 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900981 4526 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900984 4526 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900987 4526 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900991 4526 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900994 4526 flags.go:64] FLAG: --enable-server="true" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.900997 4526 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901001 4526 flags.go:64] FLAG: --event-burst="100" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901004 4526 flags.go:64] FLAG: --event-qps="50" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901007 4526 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901011 4526 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901014 4526 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901018 4526 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:16:49.907900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901021 4526 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901024 4526 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901027 4526 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901030 4526 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901035 4526 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901038 4526 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901041 4526 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901044 4526 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901047 4526 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901049 4526 flags.go:64] FLAG: --feature-gates="" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901054 4526 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901057 4526 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901060 4526 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901063 4526 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901067 4526 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901070 4526 flags.go:64] FLAG: --help="false" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901073 4526 flags.go:64] FLAG: --hostname-override="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901076 4526 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901079 4526 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901082 4526 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901085 4526 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901088 4526 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901091 4526 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:16:49.908657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901094 4526 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901097 4526 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901100 4526 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901102 4526 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901106 4526 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901108 4526 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901112 4526 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901115 4526 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901118 4526 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901121 4526 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901124 4526 flags.go:64] FLAG: --lock-file="" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901127 4526 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901130 4526 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901134 4526 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901139 4526 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901142 4526 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901145 4526 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901148 4526 flags.go:64] FLAG: --logging-format="text" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901151 4526 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901155 4526 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901158 4526 flags.go:64] FLAG: --manifest-url="" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901160 4526 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901165 4526 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901169 4526 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901173 4526 flags.go:64] FLAG: --max-pods="110" Oct 30 21:16:49.909254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901176 4526 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901179 4526 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901182 4526 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901185 4526 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901188 4526 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901191 4526 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901194 4526 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901201 4526 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901204 4526 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901207 4526 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901210 4526 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901213 4526 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901216 4526 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901220 4526 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901223 4526 flags.go:64] FLAG: --port="10250" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901226 4526 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901229 4526 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e852a63d36ab7aa5" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901232 4526 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901235 4526 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901238 4526 flags.go:64] FLAG: --register-node="true" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901245 4526 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901254 4526 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901257 4526 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901261 4526 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901263 4526 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901268 4526 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:16:49.909860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901271 4526 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901274 4526 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901276 4526 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901279 4526 flags.go:64] FLAG: --runonce="false" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901282 4526 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901285 4526 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901289 4526 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901292 4526 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901295 4526 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901298 4526 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901301 4526 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901304 4526 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901307 4526 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901310 4526 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901313 4526 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901316 4526 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901318 4526 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901321 4526 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901324 4526 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901330 4526 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901333 4526 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901336 4526 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901339 4526 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901342 4526 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901345 4526 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:16:49.910477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901348 4526 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901352 4526 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901354 4526 flags.go:64] FLAG: --v="2" Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901359 4526 flags.go:64] FLAG: --version="false" Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901363 4526 flags.go:64] FLAG: --vmodule="" Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901367 4526 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901370 4526 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901533 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901537 4526 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901540 4526 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901543 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901545 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901548 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901550 4526 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901554 4526 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901557 4526 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901559 4526 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901562 4526 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901564 4526 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901567 4526 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901569 4526 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:49.911179 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901572 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901575 4526 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901578 4526 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901580 4526 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901583 4526 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901585 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901588 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901591 4526 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901593 4526 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901596 4526 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901598 4526 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901601 4526 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901605 4526 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901608 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901610 4526 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901612 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901615 4526 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901617 4526 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901620 4526 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901622 4526 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:49.911715 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901625 4526 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901628 4526 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901632 4526 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901636 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901639 4526 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901642 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901646 4526 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901649 4526 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901651 4526 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901654 4526 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901657 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901659 4526 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901662 4526 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901664 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901666 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901669 4526 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901671 4526 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901674 4526 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901676 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901679 4526 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:49.912399 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901681 4526 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901684 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901686 4526 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901689 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901692 4526 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901695 4526 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901697 4526 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901700 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901702 4526 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901705 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901707 4526 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901710 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901712 4526 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901715 4526 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901718 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901720 4526 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901723 4526 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901725 4526 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901728 4526 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:49.913025 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901731 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901734 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901736 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901738 4526 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901741 4526 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901743 4526 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901746 4526 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901749 4526 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901753 4526 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901756 4526 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901759 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901761 4526 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901764 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901766 4526 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901769 4526 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901771 4526 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901774 4526 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901778 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901781 4526 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:49.913631 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901784 4526 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901786 4526 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901789 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901792 4526 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901795 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901798 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901800 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901803 4526 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901805 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901808 4526 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901810 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901813 4526 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901815 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901818 4526 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.901845 4526 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901886 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:49.914344 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901889 4526 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901892 4526 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901895 4526 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901898 4526 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901900 4526 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901903 4526 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901907 4526 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901911 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901915 4526 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901918 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901921 4526 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901924 4526 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901927 4526 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901930 4526 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901934 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901937 4526 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901940 4526 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901942 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901945 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901948 4526 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:49.915102 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901950 4526 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901953 4526 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901955 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901958 4526 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901960 4526 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901963 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901965 4526 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901968 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901970 4526 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901973 4526 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901975 4526 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901979 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901981 4526 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901984 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901987 4526 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901989 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901991 4526 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901994 4526 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901996 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:49.915818 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.901999 4526 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902002 4526 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902004 4526 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902007 4526 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902009 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902012 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902014 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902018 4526 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902021 4526 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902024 4526 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902026 4526 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902030 4526 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902034 4526 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902037 4526 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902039 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902042 4526 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902045 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902047 4526 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902050 4526 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:49.916400 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902052 4526 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902055 4526 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902058 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902060 4526 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902063 4526 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902065 4526 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902068 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902071 4526 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902074 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902077 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902080 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902082 4526 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902085 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902087 4526 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902090 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902092 4526 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902095 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902097 4526 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902100 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902102 4526 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:49.916952 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902107 4526 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902110 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902113 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902116 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902118 4526 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902121 4526 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902124 4526 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902126 4526 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902129 4526 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902131 4526 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902134 4526 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902137 4526 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902140 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902142 4526 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902145 4526 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902147 4526 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902150 4526 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902152 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902155 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902159 4526 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:49.917434 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902161 4526 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902164 4526 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902166 4526 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902169 4526 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902171 4526 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902174 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.902176 4526 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.907704 4526 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.907722 4526 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.907745 4526 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.907751 4526 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907820 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907825 4526 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907828 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907831 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907834 4526 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907836 4526 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907839 4526 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907842 4526 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:49.917945 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907845 4526 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907847 4526 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907850 4526 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907853 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907856 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907859 4526 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907862 4526 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907865 4526 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907868 4526 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907870 4526 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907873 4526 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907876 4526 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907878 4526 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907882 4526 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907885 4526 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907887 4526 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907890 4526 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907893 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907895 4526 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907903 4526 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:49.918416 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907906 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907908 4526 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907911 4526 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907914 4526 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907917 4526 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907919 4526 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907922 4526 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907925 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907927 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907930 4526 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907933 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907936 4526 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907938 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907941 4526 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907944 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907946 4526 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907950 4526 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907954 4526 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907963 4526 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:49.918964 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907967 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907970 4526 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907973 4526 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907975 4526 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907978 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907980 4526 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907983 4526 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907986 4526 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907989 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907991 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907994 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907997 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.907999 4526 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908002 4526 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908005 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908007 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908010 4526 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908012 4526 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908015 4526 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908017 4526 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:49.919430 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908020 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908022 4526 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908025 4526 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908028 4526 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908030 4526 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908033 4526 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908035 4526 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908038 4526 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908041 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908043 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908046 4526 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908048 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908056 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908059 4526 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908062 4526 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908064 4526 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908068 4526 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908070 4526 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908073 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:49.919976 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908076 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908079 4526 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908081 4526 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908084 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908086 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908089 4526 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908091 4526 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908094 4526 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908096 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908098 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908101 4526 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908104 4526 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908106 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908110 4526 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908113 4526 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908116 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908119 4526 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908121 4526 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908124 4526 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:49.920421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908127 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.908155 4526 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908294 4526 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908299 4526 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908302 4526 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908305 4526 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908307 4526 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908316 4526 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908319 4526 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908322 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908324 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908327 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908330 4526 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908333 4526 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908336 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908339 4526 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:49.920909 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908341 4526 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908344 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908347 4526 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908351 4526 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908355 4526 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908358 4526 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908361 4526 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908364 4526 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908367 4526 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908370 4526 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908373 4526 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908375 4526 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908378 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908381 4526 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908383 4526 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908386 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908388 4526 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908391 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908393 4526 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:49.921283 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908396 4526 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908399 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908401 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908404 4526 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908407 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908410 4526 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908413 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908417 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908419 4526 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908422 4526 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908425 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908428 4526 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908432 4526 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908435 4526 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908437 4526 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908440 4526 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908443 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908445 4526 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908448 4526 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:49.921874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908450 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908453 4526 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908455 4526 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908458 4526 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908461 4526 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908463 4526 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908466 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908468 4526 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908471 4526 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908473 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908476 4526 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908478 4526 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908481 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908501 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908505 4526 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908509 4526 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908513 4526 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908517 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908520 4526 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908525 4526 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:49.922455 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908529 4526 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908533 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908537 4526 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908539 4526 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908542 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908545 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908548 4526 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908550 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908553 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908555 4526 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908558 4526 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908560 4526 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908574 4526 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908577 4526 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908580 4526 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908582 4526 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908585 4526 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908588 4526 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908590 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908593 4526 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:49.923295 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908596 4526 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908598 4526 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908601 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908604 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908606 4526 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908609 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908611 4526 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908615 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908617 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908620 4526 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908623 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908626 4526 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908630 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908632 4526 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.908657 4526 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908690 4526 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:49.924084 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908694 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908697 4526 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908701 4526 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908705 4526 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908709 4526 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908712 4526 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908715 4526 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908718 4526 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908721 4526 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908723 4526 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908726 4526 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908728 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908731 4526 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908733 4526 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908736 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908739 4526 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908741 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908744 4526 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908747 4526 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:49.924777 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908750 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908753 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908755 4526 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908758 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908760 4526 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908763 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908766 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908768 4526 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908771 4526 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908776 4526 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908780 4526 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908784 4526 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908787 4526 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908790 4526 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908793 4526 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908796 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908798 4526 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908801 4526 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908804 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:49.925666 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908806 4526 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908809 4526 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908811 4526 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908814 4526 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908816 4526 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908819 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908822 4526 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908824 4526 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908827 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908830 4526 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908835 4526 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908837 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908840 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908842 4526 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908845 4526 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908848 4526 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908850 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908853 4526 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908855 4526 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:49.926450 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908858 4526 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908861 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908863 4526 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908868 4526 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908870 4526 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908873 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908876 4526 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908879 4526 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908881 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908884 4526 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908886 4526 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908889 4526 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908892 4526 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908894 4526 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908897 4526 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908899 4526 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908902 4526 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908905 4526 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908907 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908910 4526 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:49.927229 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908912 4526 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908915 4526 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908917 4526 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908921 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908924 4526 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908926 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908929 4526 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908932 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908934 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908937 4526 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908940 4526 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908942 4526 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908945 4526 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908948 4526 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908950 4526 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908954 4526 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908956 4526 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908959 4526 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908962 4526 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908965 4526 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:49.927760 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908968 4526 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908970 4526 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908973 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908976 4526 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908979 4526 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908982 4526 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908984 4526 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:49.908987 4526 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.909045 4526 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.910020 4526 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.910079 4526 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.910265 4526 server.go:1014] "Starting client certificate rotation" Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.910409 4526 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.910470 4526 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:11:10 +0000 UTC" deadline="2028-06-13 03:42:29.328562838 +0000 UTC" Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.910523 4526 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14190h25m39.418043661s" Oct 30 21:16:49.928306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.910825 4526 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:16:49.928824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.911867 4526 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:16:49.928824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.913429 4526 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:16:49.928824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.915576 4526 log.go:25] "Validated CRI v1 image API" Oct 30 21:16:49.928824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.916230 4526 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:16:49.928824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.917809 4526 fs.go:135] Filesystem UUIDs: map[15a7d64e-bc98-4a4f-a078-83a0e9867baa:/dev/nvme0n1p3 605eda00-e80d-442f-8dda-38a65156c993:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 21:16:49.929037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.917823 4526 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0abca98bbf95cafa478447fdfdbb0fd3faac498464e081dea3409bca2d1e0294/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0abca98bbf95cafa478447fdfdbb0fd3faac498464e081dea3409bca2d1e0294/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c8cb6d9b415aabf60731948b82bea105b919fc29a9c6ae5ac813a6586b431d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c8cb6d9b415aabf60731948b82bea105b919fc29a9c6ae5ac813a6586b431d8/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/398dacbe0c53fe2584bd4dbf1f1394a67f46504d05cc4aaf1d8bb52a2aac077d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/398dacbe0c53fe2584bd4dbf1f1394a67f46504d05cc4aaf1d8bb52a2aac077d/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46616890441e21b0b9e09f556ee9ae630a204ed29c83a1efc962c83cfbca0057/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46616890441e21b0b9e09f556ee9ae630a204ed29c83a1efc962c83cfbca0057/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66f94439360da0e59db67eed651b2ab1a5e735c801459b980e2f82a48dbf8578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66f94439360da0e59db67eed651b2ab1a5e735c801459b980e2f82a48dbf8578/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b3e40ef4b5de1ce1c7e5c1e98c1a391bb2c6efbd8d59d1374f607ae009c0c39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b3e40ef4b5de1ce1c7e5c1e98c1a391bb2c6efbd8d59d1374f607ae009c0c39/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fa9cc923e9b48a0c990cffbebd9364dad580c0a1f1b36506d01beec22114354/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fa9cc923e9b48a0c990cffbebd9364dad580c0a1f1b36506d01beec22114354/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e22c0449cb818634227da175e2b41f8d8ab4b1ce82f8342c076e115ea1e86d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e22c0449cb818634227da175e2b41f8d8ab4b1ce82f8342c076e115ea1e86d5/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8658ed5d19dc364dbc4245355d250145efff810ecb022d42245d51ef0b9bd69d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8658ed5d19dc364dbc4245355d250145efff810ecb022d42245d51ef0b9bd69d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ed777e703955cf5d60a910190303922cd5d70cc37a0808176865b0fb10a3ee1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ed777e703955cf5d60a910190303922cd5d70cc37a0808176865b0fb10a3ee1/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6f2b267e9e8df3ff3c62cd817c7221faad572c9bd8cb7033a6cfea47f48e99f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6f2b267e9e8df3ff3c62cd817c7221faad572c9bd8cb7033a6cfea47f48e99f/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc60dfa3b6d1921fa964050e1fdde4e3377d80e99643e5f43eed09ae6a6e4b5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc60dfa3b6d1921fa964050e1fdde4e3377d80e99643e5f43eed09ae6a6e4b5c/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce483c49732a81f80cbe3b1781477093ce28e60126d33ebd351d3ba0656cb445/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce483c49732a81f80cbe3b1781477093ce28e60126d33ebd351d3ba0656cb445/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edd5b9c34cf497f9ceb39247a4185820f4b18eae4ad2b7e08f259df9e10e6911/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edd5b9c34cf497f9ceb39247a4185820f4b18eae4ad2b7e08f259df9e10e6911/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f01ec999358031a5ae5f89a39cb6047b2ed173353dd4a06e4580a9414e46ae84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f01ec999358031a5ae5f89a39cb6047b2ed173353dd4a06e4580a9414e46ae84/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd01fda8ff3b47b0d7ffd7cb3f9a7f3bc1f9c8ea5a4b479d62b894b73b9ab631/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd01fda8ff3b47b0d7ffd7cb3f9a7f3bc1f9c8ea5a4b479d62b894b73b9ab631/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06b8bac2-6f8c-4a6a-9953-0573489e7846/volumes/kubernetes.io~projected/kube-api-access-vgc2k:{mountpoint:/var/lib/kubelet/pods/06b8bac2-6f8c-4a6a-9953-0573489e7846/volumes/kubernetes.io~projected/kube-api-access-vgc2k major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21d4bf6f-7b7e-45a5-a866-87c76eb414ab/volumes/kubernetes.io~projected/kube-api-access-hsxbt:{mountpoint:/var/lib/kubelet/pods/21d4bf6f-7b7e-45a5-a866-87c76eb414ab/volumes/kubernetes.io~projected/kube-api-access-hsxbt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21d4bf6f-7b7e-45a5-a866-87c76eb414ab/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/21d4bf6f-7b7e-45a5-a866-87c76eb414ab/volumes/kubernetes.io~secret/signing-key major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volumes/kubernetes.io~projected/kube-api-access-7xc7r:{mountpoint:/var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volumes/kubernetes.io~projected/kube-api-access-7xc7r major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31857e99-9c08-45ea-85da-f154ca58223d/volumes/kubernetes.io~projected/kube-api-access-f5btq:{mountpoint:/var/lib/kubelet/pods/31857e99-9c08-45ea-85da-f154ca58223d/volumes/kubernetes.io~projected/kube-api-access-f5btq major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4401a9ff-fa39-4669-b5b1-3c9f473904cf/volumes/kubernetes.io~projected/kube-api-access-q7ccw:{mountpoint:/var/lib/kubelet/pods/4401a9ff-fa39-4669-b5b1-3c9f473904cf/volumes/kubernetes.io~projected/kube-api-access-q7ccw major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62b9feff-59c5-45d5-a398-c78ad8dd590b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/62b9feff-59c5-45d5-a398-c78ad8dd590b/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b958bb-61d6-473c-bf4d-21fefe6e8675/volumes/kubernetes.io~projected/kube-api-access-55ghs:{mountpoint:/var/lib/kubelet/pods/66b958bb-61d6-473c-bf4d-21fefe6e8675/volumes/kubernetes.io~projected/kube-api-access-55ghs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df5ee35-2300-4a44-9ca3-c40545ab2866/volumes/kubernetes.io~projected/kube-api-access-48vg5:{mountpoint:/var/lib/kubelet/pods/6df5ee35-2300-4a44-9ca3-c40545ab2866/volumes/kubernetes.io~projected/kube-api-access-48vg5 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2923a9-10b7-4e7a-a828-3b8145945ba6/volumes/kubernetes.io~projected/kube-api-access-p6sr4:{mountpoint:/var/lib/kubelet/pods/6f2923a9-10b7-4e7a-a828-3b8145945ba6/volumes/kubernetes.io~projected/kube-api-access-p6sr4 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f2923a9-10b7-4e7a-a828-3b8145945ba6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6f2923a9-10b7-4e7a-a828-3b8145945ba6/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~projected/kube-api-access-6fgk6:{mountpoint:/var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~projected/kube-api-access-6fgk6 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98/volumes/kubernetes.io~projected/kube-api-access-mkwc6:{mountpoint:/var/lib/kubelet/pods/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98/volumes/kubernetes.io~projected/kube-api-access-mkwc6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8023590c-1d50-48e6-93e4-683b8368c13c/volumes/kubernetes.io~projected/kube-api-access-nvfmj:{mountpoint:/var/lib/kubelet/pods/8023590c-1d50-48e6-93e4-683b8368c13c/volumes/kubernetes.io~projected/kube-api-access-nvfmj major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/823598bc-abd8-49f6-91d9-2eb21ec1435c/volumes/kubernetes.io~projected/kube-api-access-92wb2:{mountpoint:/var/lib/kubelet/pods/823598bc-abd8-49f6-91d9-2eb21ec1435c/volumes/kubernetes.io~projected/kube-api-access-92wb2 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82f30bd6-0ef2-47f0-8f55-6f5370a098e1/volumes/kubernetes.io~projected/kube-api-access-2jv87:{mountpoint:/var/lib/kubelet/pods/82f30bd6-0ef2-47f0-8f55-6f5370a098e1/volumes/kubernetes.io~projected/kube-api-access-2jv87 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b3f89fa-e28d-4591-9fe3-0a1e06b61984/volumes/kubernetes.io~projected/kube-api-access-rq78x:{mountpoint:/var/lib/kubelet/pods/9b3f89fa-e28d-4591-9fe3-0a1e06b61984/volumes/kubernetes.io~projected/kube-api-access-rq78x major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb67e001-cef7-4f5e-9f38-2617953250d4/volumes/kubernetes.io~projected/kube-api-access-rmmrz:{mountpoint:/var/lib/kubelet/pods/cb67e001-cef7-4f5e-9f38-2617953250d4/volumes/kubernetes.io~projected/kube-api-access-rmmrz major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db28a65b-b841-4446-b062-9526c7e8139a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/db28a65b-b841-4446-b062-9526c7e8139a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb62942b-039a-4dd9-8fe2-609094d6822c/volumes/kubernetes.io~projected/kube-api-access-cmzgj:{mountpoint:/var/lib/kubelet/pods/eb62942b-039a-4dd9-8fe2-609094d6822c/volumes/kubernetes.io~projected/kube-api-access-cmzgj major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9da070b753cfac3d677884084d2d04dabe1e0f09378bee06cdbe0c8b7810e4bd/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ddfc8fbe0a8b910f4153be9020b4283191b54860d17af4370d7839830dabb70f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e4def81f6c544adf694c30c78e52312caed76713dead1f460ffc5fd581a68cd4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/77759f84826c1dccc8b8872484645a468dd93135b7a6ee5bf74b0bceb12a836c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8d948d904af3c1b04c023cc26bbcda6ab8bf701fc70a0f369f6b43d3a4972d05/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/80d146040f400764a79f50cf0e1adff81cc7717c0dbb401e348f563b8060cd4e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fda65cd7f08e669648e6ef01ce30dfcd89af6ab7d713dad7a99a4a07167e3849/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/107a44990fdd128373b9b4aad33dc37718478206a618a682648c9d1b9d09d11d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/192c3b4b589069d4c8434e90e45feb9c72517832c275ce3efc6af6d7ede74652/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/fea8e291b1352bea7c8b6971d28a94fe8d6a99c855a037324407f72efad3b45a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/f5061a97a7970047e97cb18e544c3aa9f93ad33158fae495e8c09a3cb4231a53/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/0fb8300a8a8880ffbb416ed5b9fe5abfb649631b1e3b0fce46c8395b137da761/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/509751021bf9be180331111d70096915bc52bdca8e96216020d777dfd3313cf1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9c7e2cb54b71f4c3980c923153cb8fae1d9dff3882c36da4a45712d003cccb5a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cbdcb9eb501ef9aebb89ade058e25e900ccb06728207b14e5c96e9582fd21d51/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/43d0a0cc25e40c0904281abdb17fb65c5cc8171d314ffeac27f4892151023790/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a89a958257b3b7b2b0617d712e1c81b3e96652d77ad42696bb8462b6022bba93/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/26a847edc11bae400127e27ef669ac74019544234b079c5ed7efe73d41a197b9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0033daaab45c7be52540f5e4182b9bb83350fd8159b74930c08f9c8ad862eee3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/78082cc52372ac6d903f457f760e2c5b7c0e5cea61002d59660f4373a44bc333/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/73c23adc57fdad0489925553c7050180211854c9fb74d5347ab3e21b0f21b930/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/23d9e41f23d101d1e6f4d3fc3b43a925004e1bd4440cdc41d24efc86a13bfcfb/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/85c93b0e1b2a98ad71ee0c92ea7120325d60c13977007878ad30c29064fe1048/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/05ef45da87b8ba3f401e95b932abcdc55ae215fffb8f981bbe1fdefe7cbf9ae6/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/3f4bf2a487f4842562fe7738872ff0913daaf5b3185002572d863eec3f5a510e/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/90b8d24f67d8fccc01cd467b58c8fcb55274982a48f68157f7f90506ddecbaf7/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/5746fdc7c286952c5ecf82ea21c1a954626e5c0afe3ee39399f64ffbc1a94959/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/de15a28bd2cb0b486d8be3498344f0f52d3995990326db099af27b0a48cf657d/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/fb7cdb601fd8e5e22abdb1baf6b936be70ea9e4aa352cef5baa7886b08fed64d/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e6367a771381cd61d014d335577c1c8424591ef5a71f8e8b595a429eb2dd44f5/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/ccdfd51df6d8d4691150d6fcc7e49590a2ed95925e54e408c61c78f6ae6565ec/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/05d2dac814128b3e5db3ff8bb67b6ead1dca43070782b99d1cc22f0f5dae1fbe/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/3722f58978f4801835e7d6837c86f5703bccdbbba907a55e4078e56d8f873e6c/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/ece3200b01126e6e2501b9925846a707127018d713782a807d1ea21ffd65e2c8/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/57c8db9b5322a10c9f3ce7f31af0b96526274ed0f9217be160e7708987b09f72/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0185cee53f41b2eef1a20c39cc32d7a26ea7485780f31153a07bf4dc8053b6c0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a1d10228d32f8919e2917da20c08b2850544b07826491a38308739fac864bf8d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/657a804d78fbf2d84dcc5c6ba38ce3b3c102e1a6aa7ed502c930fdebd71b0690/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ee6509e2a33b26a0d5a67e1b31b752d6e4ccf60da79a24c3ace4e4d9843858e0/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d75cca192fce9b1c83116dc727b31739ced02f2a7a8a31917fe1a343f32887fb/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4f0d533fedb9f1f0c3eb26bda4c114b84c58c6f6fe98e4cc82329bd9f47c0001/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/56b3168b6ee8e51250992a1716ec890607b735b3429d77cfa8dd6bf2fe30d288/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/35dbe0c8f65fd452ce60b021ba773abdde5d874d89adaff99ac540e744384136/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.926651 4526 manager.go:217] Machine: {Timestamp:2026-10-30 21:16:49.925841735 +0000 UTC m=+0.059214925 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093795 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29285f5664f0eb5fd8d20b01857dc5 SystemUUID:ec29285f-5664-f0eb-5fd8-d20b01857dc5 BootID:ed4f0a37-cbef-45c8-ad90-b45278863f94 Filesystems:[{Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46616890441e21b0b9e09f556ee9ae630a204ed29c83a1efc962c83cfbca0057/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8023590c-1d50-48e6-93e4-683b8368c13c/volumes/kubernetes.io~projected/kube-api-access-nvfmj DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66b958bb-61d6-473c-bf4d-21fefe6e8675/volumes/kubernetes.io~projected/kube-api-access-55ghs DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b3e40ef4b5de1ce1c7e5c1e98c1a391bb2c6efbd8d59d1374f607ae009c0c39/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21d4bf6f-7b7e-45a5-a866-87c76eb414ab/volumes/kubernetes.io~projected/kube-api-access-hsxbt DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c8cb6d9b415aabf60731948b82bea105b919fc29a9c6ae5ac813a6586b431d8/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb67e001-cef7-4f5e-9f38-2617953250d4/volumes/kubernetes.io~projected/kube-api-access-rmmrz DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc60dfa3b6d1921fa964050e1fdde4e3377d80e99643e5f43eed09ae6a6e4b5c/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4401a9ff-fa39-4669-b5b1-3c9f473904cf/volumes/kubernetes.io~projected/kube-api-access-q7ccw DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/82f30bd6-0ef2-47f0-8f55-6f5370a098e1/volumes/kubernetes.io~projected/kube-api-access-2jv87 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fa9cc923e9b48a0c990cffbebd9364dad580c0a1f1b36506d01beec22114354/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ed777e703955cf5d60a910190303922cd5d70cc37a0808176865b0fb10a3ee1/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31857e99-9c08-45ea-85da-f154ca58223d/volumes/kubernetes.io~projected/kube-api-access-f5btq DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~projected/kube-api-access-6fgk6 DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/06b8bac2-6f8c-4a6a-9953-0573489e7846/volumes/kubernetes.io~projected/kube-api-access-vgc2k DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edd5b9c34cf497f9ceb39247a4185820f4b18eae4ad2b7e08f259df9e10e6911/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98/volumes/kubernetes.io~projected/kube-api-access-mkwc6 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb62942b-039a-4dd9-8fe2-609094d6822c/volumes/kubernetes.io~projected/kube-api-access-cmzgj DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b3f89fa-e28d-4591-9fe3-0a1e06b61984/volumes/kubernetes.io~projected/kube-api-access-rq78x DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce483c49732a81f80cbe3b1781477093ce28e60126d33ebd351d3ba0656cb445/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd01fda8ff3b47b0d7ffd7cb3f9a7f3bc1f9c8ea5a4b479d62b894b73b9ab631/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2923a9-10b7-4e7a-a828-3b8145945ba6/volumes/kubernetes.io~projected/kube-api-access-p6sr4 DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66f94439360da0e59db67eed651b2ab1a5e735c801459b980e2f82a48dbf8578/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8658ed5d19dc364dbc4245355d250145efff810ecb022d42245d51ef0b9bd69d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6f2b267e9e8df3ff3c62cd817c7221faad572c9bd8cb7033a6cfea47f48e99f/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f01ec999358031a5ae5f89a39cb6047b2ed173353dd4a06e4580a9414e46ae84/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/823598bc-abd8-49f6-91d9-2eb21ec1435c/volumes/kubernetes.io~projected/kube-api-access-92wb2 DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/db28a65b-b841-4446-b062-9526c7e8139a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0abca98bbf95cafa478447fdfdbb0fd3faac498464e081dea3409bca2d1e0294/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6df5ee35-2300-4a44-9ca3-c40545ab2866/volumes/kubernetes.io~projected/kube-api-access-48vg5 DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21d4bf6f-7b7e-45a5-a866-87c76eb414ab/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8cf76d-e722-49b5-8c62-fe88eff85691/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e22c0449cb818634227da175e2b41f8d8ab4b1ce82f8342c076e115ea1e86d5/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e226384-4104-4da9-a6b9-ce21038fedc3/volumes/kubernetes.io~projected/kube-api-access-7xc7r DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f2923a9-10b7-4e7a-a828-3b8145945ba6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62b9feff-59c5-45d5-a398-c78ad8dd590b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/398dacbe0c53fe2584bd4dbf1f1394a67f46504d05cc4aaf1d8bb52a2aac077d/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0abca98bbf95caf MacAddress:1e:bc:54:21:a8:98 Speed:10000 Mtu:8901} {Name:66f94439360da0e MacAddress:fe:3c:6d:7b:2e:04 Speed:10000 Mtu:8901} {Name:6fa9cc923e9b48a MacAddress:96:87:a5:44:92:b8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:75:01:1c:c7:e1 Speed:0 Mtu:9001} {Name:br-int MacAddress:b6:f5:8b:49:28:eb Speed:0 Mtu:8901} {Name:cc60dfa3b6d1921 MacAddress:0a:d4:01:dc:b9:0e Speed:10000 Mtu:8901} {Name:ce483c49732a81f MacAddress:06:19:6b:df:ee:77 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:75:01:1c:c7:e1 Speed:0 Mtu:9001} {Name:fd01fda8ff3b47b MacAddress:4e:ed:5b:a1:52:8f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1e:de:ef:5f:be:71 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:a0:af:6c:15:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927111 4526 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927169 4526 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927468 4526 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927512 4526 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-26.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927637 4526 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927645 4526 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927656 4526 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927676 4526 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927766 4526 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.927923 4526 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.928030 4526 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.928062 4526 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.928075 4526 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.928085 4526 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.928094 4526 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.928102 4526 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.928700 4526 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:16:49.929251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.928710 4526 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:16:49.930443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.929260 4526 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:16:49.931671 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.931654 4526 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:16:49.931671 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.931670 4526 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:16:49.931842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.931806 4526 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:16:49.932032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932017 4526 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:16:49.932101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932058 4526 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:16:49.932183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932172 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:16:49.932240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932192 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:16:49.932240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932203 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:16:49.932240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932214 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:16:49.932240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932224 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:16:49.932240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932235 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:16:49.932464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932245 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:16:49.932464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932255 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:16:49.932464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932266 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:16:49.932464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932276 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:16:49.932464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932309 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:16:49.932464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932325 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:16:49.932464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932362 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:16:49.932464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.932371 4526 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:16:49.935810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.935793 4526 server.go:1263] "Started kubelet" Oct 30 21:16:49.935894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.935865 4526 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:16:49.938613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.936007 4526 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:16:49.938613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.936071 4526 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:16:49.938613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.936615 4526 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:16:49.938613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.937122 4526 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:16:49.938613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.938336 4526 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:16:49.936355 ip-10-0-128-26 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:16:49.938969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.938698 4526 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:16:49.938969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.938709 4526 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:16:49.938969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.938779 4526 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:16:49.939323 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.939148 4526 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.939456 4526 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:11:11 +0000 UTC" deadline="2028-08-08 22:00:02.621112064 +0000 UTC" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.939534 4526 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15552h43m12.681582662s" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:49.939846 4526 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940296 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06b8bac2-6f8c-4a6a-9953-0573489e7846" volumeName="kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-binary-copy" seLinuxMountContext="" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940329 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e226384-4104-4da9-a6b9-ce21038fedc3" volumeName="kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-env-overrides" seLinuxMountContext="" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940343 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e226384-4104-4da9-a6b9-ce21038fedc3" volumeName="kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-config" seLinuxMountContext="" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940354 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2923a9-10b7-4e7a-a828-3b8145945ba6" volumeName="kubernetes.io/projected/6f2923a9-10b7-4e7a-a828-3b8145945ba6-kube-api-access-p6sr4" seLinuxMountContext="" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940369 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82f30bd6-0ef2-47f0-8f55-6f5370a098e1" volumeName="kubernetes.io/empty-dir/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-tmp-dir" seLinuxMountContext="" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940380 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b3f89fa-e28d-4591-9fe3-0a1e06b61984" volumeName="kubernetes.io/projected/9b3f89fa-e28d-4591-9fe3-0a1e06b61984-kube-api-access-rq78x" seLinuxMountContext="" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940391 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31857e99-9c08-45ea-85da-f154ca58223d" volumeName="kubernetes.io/configmap/31857e99-9c08-45ea-85da-f154ca58223d-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:16:49.940395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940401 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b9feff-59c5-45d5-a398-c78ad8dd590b" volumeName="kubernetes.io/configmap/62b9feff-59c5-45d5-a398-c78ad8dd590b-konnectivity-ca" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940413 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8cf76d-e722-49b5-8c62-fe88eff85691" volumeName="kubernetes.io/projected/7b8cf76d-e722-49b5-8c62-fe88eff85691-kube-api-access-6fgk6" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940424 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8023590c-1d50-48e6-93e4-683b8368c13c" volumeName="kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940435 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8023590c-1d50-48e6-93e4-683b8368c13c" volumeName="kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940445 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8cf76d-e722-49b5-8c62-fe88eff85691" volumeName="kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-tmp" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940455 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21d4bf6f-7b7e-45a5-a866-87c76eb414ab" volumeName="kubernetes.io/secret/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-key" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940466 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b958bb-61d6-473c-bf4d-21fefe6e8675" volumeName="kubernetes.io/projected/66b958bb-61d6-473c-bf4d-21fefe6e8675-kube-api-access-55ghs" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940506 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f2923a9-10b7-4e7a-a828-3b8145945ba6" volumeName="kubernetes.io/secret/6f2923a9-10b7-4e7a-a828-3b8145945ba6-cert" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940518 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8023590c-1d50-48e6-93e4-683b8368c13c" volumeName="kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940530 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb67e001-cef7-4f5e-9f38-2617953250d4" volumeName="kubernetes.io/configmap/cb67e001-cef7-4f5e-9f38-2617953250d4-serviceca" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940541 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb67e001-cef7-4f5e-9f38-2617953250d4" volumeName="kubernetes.io/projected/cb67e001-cef7-4f5e-9f38-2617953250d4-kube-api-access-rmmrz" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940552 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db28a65b-b841-4446-b062-9526c7e8139a" volumeName="kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940571 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06b8bac2-6f8c-4a6a-9953-0573489e7846" volumeName="kubernetes.io/projected/06b8bac2-6f8c-4a6a-9953-0573489e7846-kube-api-access-vgc2k" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940582 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b958bb-61d6-473c-bf4d-21fefe6e8675" volumeName="kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-daemon-config" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940592 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31857e99-9c08-45ea-85da-f154ca58223d" volumeName="kubernetes.io/projected/31857e99-9c08-45ea-85da-f154ca58223d-kube-api-access-f5btq" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940606 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df5ee35-2300-4a44-9ca3-c40545ab2866" volumeName="kubernetes.io/projected/6df5ee35-2300-4a44-9ca3-c40545ab2866-kube-api-access-48vg5" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940617 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98" volumeName="kubernetes.io/projected/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kube-api-access-mkwc6" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940627 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb62942b-039a-4dd9-8fe2-609094d6822c" volumeName="kubernetes.io/configmap/eb62942b-039a-4dd9-8fe2-609094d6822c-config-volume" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940645 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06b8bac2-6f8c-4a6a-9953-0573489e7846" volumeName="kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940656 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e226384-4104-4da9-a6b9-ce21038fedc3" volumeName="kubernetes.io/secret/2e226384-4104-4da9-a6b9-ce21038fedc3-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:16:49.940951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.940666 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823598bc-abd8-49f6-91d9-2eb21ec1435c" volumeName="kubernetes.io/projected/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-api-access-92wb2" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941135 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb62942b-039a-4dd9-8fe2-609094d6822c" volumeName="kubernetes.io/empty-dir/eb62942b-039a-4dd9-8fe2-609094d6822c-tmp-dir" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941166 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8023590c-1d50-48e6-93e4-683b8368c13c" volumeName="kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941269 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06b8bac2-6f8c-4a6a-9953-0573489e7846" volumeName="kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941281 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21d4bf6f-7b7e-45a5-a866-87c76eb414ab" volumeName="kubernetes.io/configmap/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-cabundle" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941337 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e226384-4104-4da9-a6b9-ce21038fedc3" volumeName="kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941354 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62b9feff-59c5-45d5-a398-c78ad8dd590b" volumeName="kubernetes.io/secret/62b9feff-59c5-45d5-a398-c78ad8dd590b-agent-certs" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941367 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b958bb-61d6-473c-bf4d-21fefe6e8675" volumeName="kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-cni-binary-copy" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941432 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="823598bc-abd8-49f6-91d9-2eb21ec1435c" volumeName="kubernetes.io/empty-dir/823598bc-abd8-49f6-91d9-2eb21ec1435c-data-volume" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941445 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82f30bd6-0ef2-47f0-8f55-6f5370a098e1" volumeName="kubernetes.io/projected/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-kube-api-access-2jv87" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941456 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21d4bf6f-7b7e-45a5-a866-87c76eb414ab" volumeName="kubernetes.io/projected/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-kube-api-access-hsxbt" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941472 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e226384-4104-4da9-a6b9-ce21038fedc3" volumeName="kubernetes.io/projected/2e226384-4104-4da9-a6b9-ce21038fedc3-kube-api-access-7xc7r" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941505 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4401a9ff-fa39-4669-b5b1-3c9f473904cf" volumeName="kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941524 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8cf76d-e722-49b5-8c62-fe88eff85691" volumeName="kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-tuned" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941536 4526 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb62942b-039a-4dd9-8fe2-609094d6822c" volumeName="kubernetes.io/projected/eb62942b-039a-4dd9-8fe2-609094d6822c-kube-api-access-cmzgj" seLinuxMountContext="" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941549 4526 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:16:49.941859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.941556 4526 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:16:49.949470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.946304 4526 factory.go:153] Registering CRI-O factory Oct 30 21:16:49.949759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.946327 4526 factory.go:223] Registration of the crio container factory successfully Oct 30 21:16:49.949840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.949821 4526 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:16:49.949840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.949833 4526 factory.go:55] Registering systemd factory Oct 30 21:16:49.949931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.949843 4526 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:16:49.949931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.949862 4526 factory.go:103] Registering Raw factory Oct 30 21:16:49.949931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.949877 4526 manager.go:1199] Started watching for new ooms in manager Oct 30 21:16:49.950179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.950166 4526 manager.go:319] Starting recovery of all containers Oct 30 21:16:49.975868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.975811 4526 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:16:49.977245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.977220 4526 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:16:49.977245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.977243 4526 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:16:49.977422 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.977266 4526 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:16:49.977422 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:49.977307 4526 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:16:49.991503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.991452 4526 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="5021aa801c4085afd539fb1b26ab584b12ca18e868b8eb57d0df3433d517860f" exitCode=0 Oct 30 21:16:49.991653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.991540 4526 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="87e6e971180955af2746255e903a357e3a99f67d34c5d4c202901d5f011be932" exitCode=0 Oct 30 21:16:49.991653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.991553 4526 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="af1fc20218c6ff619d1f8f91a1342e1256d4f58df2395f9c4bf98a485af19d74" exitCode=0 Oct 30 21:16:49.991653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.991562 4526 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="c1a18d7c2eb01268fb333d4f1d0b793a72724625eefe655a7cfa4f85a4c4947d" exitCode=0 Oct 30 21:16:49.991653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.991571 4526 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="429977a5adbd4d1d8a953a7bbf757f86acdc1183544cd5fd250eca4d98977567" exitCode=0 Oct 30 21:16:49.991653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:49.991581 4526 generic.go:356] "Generic (PLEG): container finished" podID="06b8bac2-6f8c-4a6a-9953-0573489e7846" containerID="c39863edac5c872a76e1b10714014e19ee0cde093637fd2a9cd1a451fa4fddaa" exitCode=0 Oct 30 21:16:50.015112 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.015068 4526 generic.go:356] "Generic (PLEG): container finished" podID="2e8b937f14ab0703a8793557804f6cae" containerID="29e045af7c731199c539bf0dd225061d1358d0744e95210a7c3faa02cadd7b03" exitCode=0 Oct 30 21:16:50.036683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.036651 4526 manager.go:324] Recovery completed Oct 30 21:16:50.039973 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:50.039938 4526 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" Oct 30 21:16:50.052463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.052443 4526 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:50.053783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.053761 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:50.053893 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.053802 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:50.053893 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.053819 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:50.055305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055290 4526 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:16:50.055305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055302 4526 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:16:50.055305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055305 4526 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:50.055470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055317 4526 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:16:50.055470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055440 4526 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:16:50.055470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055452 4526 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:16:50.055592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055549 4526 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:16:50.055592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055555 4526 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:16:50.055592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055561 4526 policy_none.go:50] "Start" Oct 30 21:16:50.055592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055569 4526 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:16:50.055592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055578 4526 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:16:50.055716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055699 4526 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:16:50.055716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055705 4526 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:16:50.055716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.055711 4526 policy_none.go:44] "Start" Oct 30 21:16:50.056548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.056532 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:50.056623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.056570 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:50.056623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.056581 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:50.059960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.059933 4526 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:16:50.062857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.062842 4526 manager.go:341] "Starting Device Plugin manager" Oct 30 21:16:50.062944 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:50.062868 4526 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:16:50.062944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.062878 4526 server.go:87] "Starting device plugin registration server" Oct 30 21:16:50.063139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.063127 4526 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:16:50.063172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.063140 4526 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:16:50.063264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.063247 4526 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:16:50.063390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.063355 4526 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:16:50.063522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.063398 4526 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:16:50.063522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.063405 4526 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:16:50.063634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.063478 4526 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:16:50.063385532Z","UUID":"4f682116-2490-4c6b-998b-06dbdc1dd787","Handler":null,"Name":"","Endpoint":""} Oct 30 21:16:50.064732 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:50.064709 4526 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:16:50.064823 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:50.064753 4526 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" Oct 30 21:16:50.065048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.065031 4526 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:16:50.065159 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.065053 4526 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:16:50.078739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.078222 4526 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal"] Oct 30 21:16:50.078835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.078820 4526 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:50.079791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.079747 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:50.079879 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.079807 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:50.079879 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.079821 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:50.080128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.080115 4526 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:50.080264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.080239 4526 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:50.081017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.080998 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:50.081017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081027 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:50.081124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081000 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:50.081124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081040 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:50.081124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081072 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:50.081124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081088 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:50.081270 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:50.081217 4526 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.081302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081270 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"2e8b937f14ab0703a8793557804f6cae","Type":"ContainerStarted","Data":"afd83a8e0f80c9f479d27d629198ee7c69dce910d29999658a5b9c66ff2fbcf9"} Oct 30 21:16:50.081330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081300 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"2e8b937f14ab0703a8793557804f6cae","Type":"ContainerDied","Data":"29e045af7c731199c539bf0dd225061d1358d0744e95210a7c3faa02cadd7b03"} Oct 30 21:16:50.081330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081313 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"2e8b937f14ab0703a8793557804f6cae","Type":"ContainerStarted","Data":"edd5b9c34cf497f9ceb39247a4185820f4b18eae4ad2b7e08f259df9e10e6911"} Oct 30 21:16:50.081387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081340 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"a4207178a950303c256d4f333088d96f","Type":"ContainerStarted","Data":"147820bfa14f1419564ae6f75136e929a33ec5cf13e2d397b08896c729e7787f"} Oct 30 21:16:50.081387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081348 4526 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:50.081454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.081351 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" event={"ID":"a4207178a950303c256d4f333088d96f","Type":"ContainerStarted","Data":"a6f2b267e9e8df3ff3c62cd817c7221faad572c9bd8cb7033a6cfea47f48e99f"} Oct 30 21:16:50.082799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.082783 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:50.082871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.082810 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:50.082871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.082820 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:50.082941 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:50.082930 4526 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-26.us-west-1.compute.internal\" not found" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.132834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.132725 4526 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:50.132834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.132825 4526 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:50.132978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.132962 4526 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:16:50.134042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.134023 4526 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:50.141012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.140996 4526 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.164190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.164170 4526 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:50.165228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.165205 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:50.165320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.165234 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:50.165320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.165244 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:50.165320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.165282 4526 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.184109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.184071 4526 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:50.198738 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.198717 4526 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:16:50.198903 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:50.198778 4526 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.198903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.198795 4526 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.233395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.233370 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4207178a950303c256d4f333088d96f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"a4207178a950303c256d4f333088d96f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.233395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.233398 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.233585 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.233419 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.257726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.257705 4526 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:16:50.257840 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:50.257748 4526 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.282638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.282615 4526 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.282745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.282675 4526 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.283245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.283227 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeNotReady" Oct 30 21:16:50.283331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.283260 4526 setters.go:546] "Node became not ready" node="ip-10-0-128-26.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:16:50Z","lastTransitionTime":"2026-10-30T21:16:50Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:16:50.295398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.295376 4526 nodeinfomanager.go:471] existing CSINode "ip-10-0-128-26.us-west-1.compute.internal" is owned by different node (oldNodeID="ed29bb77-a4f2-4fab-8bfd-b53094700d3b", newNodeID=""), cleaning up... Oct 30 21:16:50.333819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.333787 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4207178a950303c256d4f333088d96f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"a4207178a950303c256d4f333088d96f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.333819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.333817 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.334012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.333865 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.334012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.333904 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.334012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.333913 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4207178a950303c256d4f333088d96f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"a4207178a950303c256d4f333088d96f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.334012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.333946 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e8b937f14ab0703a8793557804f6cae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" (UID: \"2e8b937f14ab0703a8793557804f6cae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:50.929106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:50.929077 4526 apiserver.go:52] "Watching apiserver" Oct 30 21:16:51.031395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.031367 4526 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:16:51.031841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.031819 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-qc6qf","openshift-insights/insights-runtime-extractor-4z7zj","openshift-multus/network-metrics-daemon-55bl6","openshift-multus/multus-additional-cni-plugins-84tpc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n","openshift-multus/multus-kd826","kube-system/konnectivity-agent-tljvj","openshift-image-registry/node-ca-xlrvq","openshift-insights/periodic-gathering-cpc25-sxdlt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z","openshift-dns/dns-default-chx8w","openshift-dns/node-resolver-zpmnj","openshift-network-operator/iptables-alerter-6r5rg","openshift-service-ca/service-ca-5fffc97bdf-sg4nr","openshift-cluster-node-tuning-operator/tuned-n7bs2","openshift-network-diagnostics/network-check-target-79bgk","kube-system/global-pull-secret-syncer-bbwl8","openshift-ovn-kubernetes/ovnkube-node-sk6xz"] Oct 30 21:16:51.032472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.032456 4526 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:51.032841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.032785 4526 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:51.033328 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.033311 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:51.041176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.041153 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.042752 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.042736 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.087735 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.087710 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:16:51.087883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.087866 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-msgtv\"" Oct 30 21:16:51.088007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.087989 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:16:51.088038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.087995 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lps2\"" Oct 30 21:16:51.091660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.091646 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ztz2s\"" Oct 30 21:16:51.092041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092019 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:16:51.092464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092443 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ks46b\"" Oct 30 21:16:51.092591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092463 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:16:51.092591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092472 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kknvl\"" Oct 30 21:16:51.092591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092530 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8g8v4\"" Oct 30 21:16:51.092591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092571 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p2m87\"" Oct 30 21:16:51.092591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092576 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fp6d9\"" Oct 30 21:16:51.092838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092572 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5jmhs\"" Oct 30 21:16:51.092838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092634 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrcct\"" Oct 30 21:16:51.092838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092759 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:16:51.093121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.092973 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.093121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093003 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:16:51.093121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093015 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.093121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093019 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:16:51.093121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093017 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.093253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093186 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:16:51.093253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093228 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.093303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093252 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.093303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093268 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.093303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093267 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.093303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093289 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:16:51.093408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093308 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:16:51.093408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093318 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:16:51.093408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093327 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.093408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093361 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.093408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093360 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:16:51.093408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093392 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:16:51.093596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.093426 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.099231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.099213 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6llv\"" Oct 30 21:16:51.099330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.099241 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jmnhg\"" Oct 30 21:16:51.099330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.099285 4526 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:16:51.099420 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:51.099331 4526 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:51.099672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.099650 4526 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:16:51.099732 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:51.099691 4526 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-26.us-west-1.compute.internal" Oct 30 21:16:51.139897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.139864 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.140034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.139907 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-device-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.140034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.139926 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-kubernetes\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.140034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.139990 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-tuned\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.140034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140025 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-dbus\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:51.140201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140054 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5btq\" (UniqueName: \"kubernetes.io/projected/31857e99-9c08-45ea-85da-f154ca58223d-kube-api-access-f5btq\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:51.140201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140180 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-tuned\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.140365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140351 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.140400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140374 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-node-log\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.140400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140389 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-sys-fs\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.140400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140393 4526 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:16:51.140548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140408 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62942b-039a-4dd9-8fe2-609094d6822c-config-volume\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.140548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140516 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgc2k\" (UniqueName: \"kubernetes.io/projected/06b8bac2-6f8c-4a6a-9953-0573489e7846-kube-api-access-vgc2k\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.140631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140569 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb67e001-cef7-4f5e-9f38-2617953250d4-host\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:51.140631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140594 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-systemd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.140685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140643 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:51.140685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140671 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-hostroot\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.140750 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140723 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-ovn\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.140795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140763 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-env-overrides\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.140826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140794 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.140826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140819 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysconfig\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.140911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140844 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.140911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140867 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-system-cni-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.140985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140911 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-bin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.140985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140949 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-conf-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.140985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140975 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-log-socket\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.141065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.140982 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-env-overrides\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.141065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141002 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-multus-certs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.141065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141035 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-var-lib-kubelet\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.141065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141063 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-tmp\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.141216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141108 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-systemd-units\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.141216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141133 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b8cf76d-e722-49b5-8c62-fe88eff85691-tmp\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.141216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141150 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-conf\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.141216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141185 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62b9feff-59c5-45d5-a398-c78ad8dd590b-agent-certs\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:51.141216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141203 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:51.141450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141219 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31857e99-9c08-45ea-85da-f154ca58223d-iptables-alerter-script\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:51.141450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141259 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-hosts-file\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:51.141450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141293 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-bin\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.141450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141337 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.141450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141359 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-var-lib-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.141450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141376 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.141450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141393 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.141818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141466 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-multus\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.141818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141534 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/62b9feff-59c5-45d5-a398-c78ad8dd590b-agent-certs\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:51.141818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141573 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-kubelet-config\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:51.141818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141617 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb67e001-cef7-4f5e-9f38-2617953250d4-serviceca\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:51.141818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141657 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-netns\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.141818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141717 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-config\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.141818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141735 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.141818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141751 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141854 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vg5\" (UniqueName: \"kubernetes.io/projected/6df5ee35-2300-4a44-9ca3-c40545ab2866-kube-api-access-48vg5\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141886 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-lib-modules\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141930 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-host\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141959 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-binary-copy\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141980 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-netns\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.141994 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-config\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142001 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62b9feff-59c5-45d5-a398-c78ad8dd590b-konnectivity-ca\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142030 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142054 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142077 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-socket-dir-parent\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142136 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmmrz\" (UniqueName: \"kubernetes.io/projected/cb67e001-cef7-4f5e-9f38-2617953250d4-kube-api-access-rmmrz\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:51.142168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142158 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06b8bac2-6f8c-4a6a-9953-0573489e7846-cni-binary-copy\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142176 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-script-lib\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142218 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/62b9feff-59c5-45d5-a398-c78ad8dd590b-konnectivity-ca\") pod \"konnectivity-agent-tljvj\" (UID: \"62b9feff-59c5-45d5-a398-c78ad8dd590b\") " pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142249 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzgj\" (UniqueName: \"kubernetes.io/projected/eb62942b-039a-4dd9-8fe2-609094d6822c-kube-api-access-cmzgj\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142270 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-etc-kubernetes\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142313 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142330 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-modprobe-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142347 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-cni-binary-copy\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142372 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e226384-4104-4da9-a6b9-ce21038fedc3-ovnkube-script-lib\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142374 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-kubelet\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142432 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142505 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-systemd\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142548 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-run\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142588 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgk6\" (UniqueName: \"kubernetes.io/projected/7b8cf76d-e722-49b5-8c62-fe88eff85691-kube-api-access-6fgk6\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.142625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142620 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-cnibin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142669 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-kubelet\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142706 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-cni-binary-copy\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142734 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-tmp-dir\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142764 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-etc-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142799 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-tmp-dir\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142798 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-socket-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142828 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-registration-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142850 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-sys\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142872 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-cnibin\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142905 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-system-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142939 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62942b-039a-4dd9-8fe2-609094d6822c-tmp-dir\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.142975 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jv87\" (UniqueName: \"kubernetes.io/projected/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-kube-api-access-2jv87\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:51.143015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143006 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-slash\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143036 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e226384-4104-4da9-a6b9-ce21038fedc3-ovn-node-metrics-cert\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143059 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xc7r\" (UniqueName: \"kubernetes.io/projected/2e226384-4104-4da9-a6b9-ce21038fedc3-kube-api-access-7xc7r\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143078 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb62942b-039a-4dd9-8fe2-609094d6822c-tmp-dir\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143135 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kube-api-access-mkwc6\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143172 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-k8s-cni-cncf-io\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143226 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143232 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e226384-4104-4da9-a6b9-ce21038fedc3-ovn-node-metrics-cert\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143251 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31857e99-9c08-45ea-85da-f154ca58223d-host-slash\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143272 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-netd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143363 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-daemon-config\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.143471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143387 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ghs\" (UniqueName: \"kubernetes.io/projected/66b958bb-61d6-473c-bf4d-21fefe6e8675-kube-api-access-55ghs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.143819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143538 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/db28a65b-b841-4446-b062-9526c7e8139a-original-pull-secret\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:51.143819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.143557 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-daemon-config\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.187140 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187069 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:16:51.187274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187246 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:16:51.187344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187293 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:16:51.187395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187380 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7dj2\"" Oct 30 21:16:51.187445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187400 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:16:51.187445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187404 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k4xtg\"" Oct 30 21:16:51.187523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187479 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wvjk\"" Oct 30 21:16:51.187552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187545 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42q2c\"" Oct 30 21:16:51.187630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187618 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pc6p4\"" Oct 30 21:16:51.187694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.187685 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ptpxj\"" Oct 30 21:16:51.192106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192091 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.192175 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192120 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:16:51.192215 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192178 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.192244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192237 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:16:51.192472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192456 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.192560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192546 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.192794 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192779 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:16:51.192860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192779 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:16:51.192860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192784 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.192860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192793 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.192860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192816 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.192860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192818 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.192860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192831 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.192860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192826 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.193105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192951 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.193105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192957 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.193105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.192981 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:16:51.193105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.193022 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:16:51.193105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.193078 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:16:51.193105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.193085 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:16:51.194271 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.194256 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:16:51.195109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.195089 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5btq\" (UniqueName: \"kubernetes.io/projected/31857e99-9c08-45ea-85da-f154ca58223d-kube-api-access-f5btq\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:51.195753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.195735 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgc2k\" (UniqueName: \"kubernetes.io/projected/06b8bac2-6f8c-4a6a-9953-0573489e7846-kube-api-access-vgc2k\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.195969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.195952 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7ccw\" (UniqueName: \"kubernetes.io/projected/4401a9ff-fa39-4669-b5b1-3c9f473904cf-kube-api-access-q7ccw\") pod \"network-check-target-79bgk\" (UID: \"4401a9ff-fa39-4669-b5b1-3c9f473904cf\") " pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:16:51.196286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.196268 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vg5\" (UniqueName: \"kubernetes.io/projected/6df5ee35-2300-4a44-9ca3-c40545ab2866-kube-api-access-48vg5\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:51.197156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.197129 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgk6\" (UniqueName: \"kubernetes.io/projected/7b8cf76d-e722-49b5-8c62-fe88eff85691-kube-api-access-6fgk6\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.197250 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.197115 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzgj\" (UniqueName: \"kubernetes.io/projected/eb62942b-039a-4dd9-8fe2-609094d6822c-kube-api-access-cmzgj\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.197309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.197260 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmmrz\" (UniqueName: \"kubernetes.io/projected/cb67e001-cef7-4f5e-9f38-2617953250d4-kube-api-access-rmmrz\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:51.197395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.197382 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ghs\" (UniqueName: \"kubernetes.io/projected/66b958bb-61d6-473c-bf4d-21fefe6e8675-kube-api-access-55ghs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.197568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.197552 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwc6\" (UniqueName: \"kubernetes.io/projected/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kube-api-access-mkwc6\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.197786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.197768 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jv87\" (UniqueName: \"kubernetes.io/projected/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-kube-api-access-2jv87\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:51.198076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.198058 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xc7r\" (UniqueName: \"kubernetes.io/projected/2e226384-4104-4da9-a6b9-ce21038fedc3-kube-api-access-7xc7r\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.200737 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.200720 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62942b-039a-4dd9-8fe2-609094d6822c-config-volume\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.201836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.201820 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb67e001-cef7-4f5e-9f38-2617953250d4-serviceca\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:51.201898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.201843 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31857e99-9c08-45ea-85da-f154ca58223d-iptables-alerter-script\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:51.243804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.243782 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-systemd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.243885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.243806 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:51.243885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.243821 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-hostroot\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.243885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.243837 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:51.243885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.243851 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-key\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:51.244068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.243897 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-systemd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.244068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.243913 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-hostroot\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.244068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.243937 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-ovn\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.244068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244020 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.244068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244030 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-ovn\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.244068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244032 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:51.244068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244048 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysconfig\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244053 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-key\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244075 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244118 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-system-cni-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244076 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-etc-selinux\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244120 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysconfig\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244144 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-bin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244152 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-system-cni-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244186 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-conf-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244189 4526 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244233 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-log-socket\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244237 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-conf-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244251 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-multus-certs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244275 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-var-lib-kubelet\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244277 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-log-socket\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244292 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-systemd-units\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244326 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-multus-certs\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244336 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-systemd-units\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244352 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-conf\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244363 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-var-lib-kubelet\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244202 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-bin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.244373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244377 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-hosts-file\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244402 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-bin\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244420 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244431 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82f30bd6-0ef2-47f0-8f55-6f5370a098e1-hosts-file\") pod \"node-resolver-zpmnj\" (UID: \"82f30bd6-0ef2-47f0-8f55-6f5370a098e1\") " pod="openshift-dns/node-resolver-zpmnj" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244442 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-var-lib-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244450 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-bin\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244476 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-tuning-conf-dir\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244476 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-conf\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244575 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-var-lib-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244899 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-multus\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244925 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-kubelet-config\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244937 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-cni-multus\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244947 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-netns\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244976 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-netns\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.244980 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-lib-modules\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245011 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-kubelet-config\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245359 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-host\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245405 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-netns\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245434 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.245553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245466 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245578 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245642 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-host\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245689 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-netns\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245748 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245846 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-socket-dir-parent\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245903 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq78x\" (UniqueName: \"kubernetes.io/projected/9b3f89fa-e28d-4591-9fe3-0a1e06b61984-kube-api-access-rq78x\") pod \"migrator-66df5d496b-7tz9z\" (UID: \"9b3f89fa-e28d-4591-9fe3-0a1e06b61984\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245932 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.245991 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-etc-kubernetes\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246075 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/823598bc-abd8-49f6-91d9-2eb21ec1435c-data-volume\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246114 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2923a9-10b7-4e7a-a828-3b8145945ba6-cert\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246167 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246197 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-modprobe-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246247 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246279 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxbt\" (UniqueName: \"kubernetes.io/projected/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-kube-api-access-hsxbt\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246326 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-kubelet\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246355 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246396 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-systemd\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246424 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-run\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246472 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-cnibin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246553 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-kubelet\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246605 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-etc-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246630 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-socket-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246677 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-multus-socket-dir-parent\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246678 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-registration-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246728 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-sys\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246748 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-registration-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246763 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-cnibin\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.246778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246789 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-system-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246822 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-slash\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246861 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-k8s-cni-cncf-io\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246897 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31857e99-9c08-45ea-85da-f154ca58223d-host-slash\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246926 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-netd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246964 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823598bc-abd8-49f6-91d9-2eb21ec1435c-crio-socket\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.246999 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247036 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6sr4\" (UniqueName: \"kubernetes.io/projected/6f2923a9-10b7-4e7a-a828-3b8145945ba6-kube-api-access-p6sr4\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247074 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-cabundle\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247074 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5ee35-2300-4a44-9ca3-c40545ab2866-metrics-certs\") pod \"network-metrics-daemon-55bl6\" (UID: \"6df5ee35-2300-4a44-9ca3-c40545ab2866\") " pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247093 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb62942b-039a-4dd9-8fe2-609094d6822c-metrics-tls\") pod \"dns-default-chx8w\" (UID: \"eb62942b-039a-4dd9-8fe2-609094d6822c\") " pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247132 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247153 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247169 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-device-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247175 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247200 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-kubernetes\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247230 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-dbus\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247250 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-sys\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247264 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247317 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-kubelet\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247326 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-system-cni-dir\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247264 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247343 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-systemd\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247370 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-etc-kubernetes\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247376 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247420 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-var-lib-kubelet\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247434 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-run\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247474 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247480 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-etc-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247539 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-node-log\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247563 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-sys-fs\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247589 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-socket-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247594 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f2923a9-10b7-4e7a-a828-3b8145945ba6-cert\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247606 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-run-openvswitch\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247692 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-cnibin\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247697 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-slash\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247702 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-modprobe-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247729 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-lib-modules\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247784 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06b8bac2-6f8c-4a6a-9953-0573489e7846-cnibin\") pod \"multus-additional-cni-plugins-84tpc\" (UID: \"06b8bac2-6f8c-4a6a-9953-0573489e7846\") " pod="openshift-multus/multus-additional-cni-plugins-84tpc" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247800 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-sysctl-d\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247819 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb67e001-cef7-4f5e-9f38-2617953250d4-host\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247901 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-cni-netd\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247943 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.247981 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-sys-fs\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248017 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-signing-cabundle\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248029 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31857e99-9c08-45ea-85da-f154ca58223d-host-slash\") pod \"iptables-alerter-6r5rg\" (UID: \"31857e99-9c08-45ea-85da-f154ca58223d\") " pod="openshift-network-operator/iptables-alerter-6r5rg" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248079 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/db28a65b-b841-4446-b062-9526c7e8139a-dbus\") pod \"global-pull-secret-syncer-bbwl8\" (UID: \"db28a65b-b841-4446-b062-9526c7e8139a\") " pod="kube-system/global-pull-secret-syncer-bbwl8" Oct 30 21:16:51.248050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248085 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e226384-4104-4da9-a6b9-ce21038fedc3-node-log\") pod \"ovnkube-node-sk6xz\" (UID: \"2e226384-4104-4da9-a6b9-ce21038fedc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.250294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248090 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98-device-dir\") pod \"aws-ebs-csi-driver-node-5q82n\" (UID: \"7f9e20d3-04ff-4a1c-a5e2-4accc5aedd98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5q82n" Oct 30 21:16:51.250294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248119 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66b958bb-61d6-473c-bf4d-21fefe6e8675-host-run-k8s-cni-cncf-io\") pod \"multus-kd826\" (UID: \"66b958bb-61d6-473c-bf4d-21fefe6e8675\") " pod="openshift-multus/multus-kd826" Oct 30 21:16:51.250294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248160 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92wb2\" (UniqueName: \"kubernetes.io/projected/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-api-access-92wb2\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.250294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248170 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/823598bc-abd8-49f6-91d9-2eb21ec1435c-data-volume\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.250294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248174 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8cf76d-e722-49b5-8c62-fe88eff85691-etc-kubernetes\") pod \"tuned-n7bs2\" (UID: \"7b8cf76d-e722-49b5-8c62-fe88eff85691\") " pod="openshift-cluster-node-tuning-operator/tuned-n7bs2" Oct 30 21:16:51.250294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.248353 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb67e001-cef7-4f5e-9f38-2617953250d4-host\") pod \"node-ca-xlrvq\" (UID: \"cb67e001-cef7-4f5e-9f38-2617953250d4\") " pod="openshift-image-registry/node-ca-xlrvq" Oct 30 21:16:51.301278 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.301242 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxbt\" (UniqueName: \"kubernetes.io/projected/21d4bf6f-7b7e-45a5-a866-87c76eb414ab-kube-api-access-hsxbt\") pod \"service-ca-5fffc97bdf-sg4nr\" (UID: \"21d4bf6f-7b7e-45a5-a866-87c76eb414ab\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sg4nr" Oct 30 21:16:51.301661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.301638 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq78x\" (UniqueName: \"kubernetes.io/projected/9b3f89fa-e28d-4591-9fe3-0a1e06b61984-kube-api-access-rq78x\") pod \"migrator-66df5d496b-7tz9z\" (UID: \"9b3f89fa-e28d-4591-9fe3-0a1e06b61984\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7tz9z" Oct 30 21:16:51.301888 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.301870 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj\") pod \"periodic-gathering-cpc25-sxdlt\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:16:51.301954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.301933 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6sr4\" (UniqueName: \"kubernetes.io/projected/6f2923a9-10b7-4e7a-a828-3b8145945ba6-kube-api-access-p6sr4\") pod \"ingress-canary-qc6qf\" (UID: \"6f2923a9-10b7-4e7a-a828-3b8145945ba6\") " pod="openshift-ingress-canary/ingress-canary-qc6qf" Oct 30 21:16:51.302221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.302206 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92wb2\" (UniqueName: \"kubernetes.io/projected/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-api-access-92wb2\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.334048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.334024 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-55bl6" Oct 30 21:16:51.342746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.342719 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:51.349032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.349012 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823598bc-abd8-49f6-91d9-2eb21ec1435c-crio-socket\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.349123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.349037 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.349123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.349057 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.349123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.349119 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/823598bc-abd8-49f6-91d9-2eb21ec1435c-crio-socket\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.349282 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:51.349153 4526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:16:51.349282 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:51.349224 4526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:51.849198093 +0000 UTC m=+1.982571284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:16:51.349614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.349596 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/823598bc-abd8-49f6-91d9-2eb21ec1435c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.451935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.451868 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:51.748580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.748506 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qc6qf_6f2923a9-10b7-4e7a-a828-3b8145945ba6/serve-healthcheck-canary/0.log" Oct 30 21:16:51.813176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.813149 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-55bl6"] Oct 30 21:16:51.816013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.815994 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-chx8w"] Oct 30 21:16:51.817553 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:51.817527 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df5ee35_2300_4a44_9ca3_c40545ab2866.slice/crio-8abc38d178a158b18aed1757c524c677d6e122888240ffcdfdfa0bdc2d8494ca WatchSource:0}: Error finding container 8abc38d178a158b18aed1757c524c677d6e122888240ffcdfdfa0bdc2d8494ca: Status 404 returned error can't find the container with id 8abc38d178a158b18aed1757c524c677d6e122888240ffcdfdfa0bdc2d8494ca Oct 30 21:16:51.819658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.819643 4526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:16:51.852663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.852642 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:51.852772 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:51.852757 4526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:16:51.852828 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:51.852819 4526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:52.852804925 +0000 UTC m=+2.986178104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:16:51.867503 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:51.867464 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb62942b_039a_4dd9_8fe2_609094d6822c.slice/crio-8251aaddc4e8a965141e32ea126af5e25d314504bffcbe187d73cf69457f7667 WatchSource:0}: Error finding container 8251aaddc4e8a965141e32ea126af5e25d314504bffcbe187d73cf69457f7667: Status 404 returned error can't find the container with id 8251aaddc4e8a965141e32ea126af5e25d314504bffcbe187d73cf69457f7667 Oct 30 21:16:51.947860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:51.947832 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-7tz9z_9b3f89fa-e28d-4591-9fe3-0a1e06b61984/migrator/0.log" Oct 30 21:16:52.035330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:52.035296 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55bl6" event={"ID":"6df5ee35-2300-4a44-9ca3-c40545ab2866","Type":"ContainerStarted","Data":"8abc38d178a158b18aed1757c524c677d6e122888240ffcdfdfa0bdc2d8494ca"} Oct 30 21:16:52.036410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:52.036375 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chx8w" event={"ID":"eb62942b-039a-4dd9-8fe2-609094d6822c","Type":"ContainerStarted","Data":"8251aaddc4e8a965141e32ea126af5e25d314504bffcbe187d73cf69457f7667"} Oct 30 21:16:52.861932 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:52.861824 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:52.862114 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:52.862041 4526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:16:52.862205 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:52.862156 4526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:54.862135505 +0000 UTC m=+4.995508697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:16:54.043932 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:54.043902 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55bl6" event={"ID":"6df5ee35-2300-4a44-9ca3-c40545ab2866","Type":"ContainerStarted","Data":"0a5aa2a97a4e0056433aff1023add2db9329acc0aa124baf6a9cd556dd18ec89"} Oct 30 21:16:54.044277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:54.043952 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-55bl6" event={"ID":"6df5ee35-2300-4a44-9ca3-c40545ab2866","Type":"ContainerStarted","Data":"50de9bb9fe0f5e14cd6db4f25715deea148a58a6e4675477028734680fa0547d"} Oct 30 21:16:54.885459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:54.885415 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:54.885637 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:54.885557 4526 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:16:54.885637 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:16:54.885614 4526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls podName:823598bc-abd8-49f6-91d9-2eb21ec1435c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:58.885599657 +0000 UTC m=+9.018972833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4z7zj" (UID: "823598bc-abd8-49f6-91d9-2eb21ec1435c") : secret "insights-runtime-extractor-tls" not found Oct 30 21:16:55.049361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:55.049322 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chx8w" event={"ID":"eb62942b-039a-4dd9-8fe2-609094d6822c","Type":"ContainerStarted","Data":"6cca6482a88e98cc1de294825ff068f372d570ebdbf6bc634a41e5d11e36c2b7"} Oct 30 21:16:55.049742 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:55.049388 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chx8w" event={"ID":"eb62942b-039a-4dd9-8fe2-609094d6822c","Type":"ContainerStarted","Data":"8749b6c022e021f32fe077ace751ee8accaf55f7faa806d0d40539610e4d5e3c"} Oct 30 21:16:55.049742 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:55.049607 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-chx8w" Oct 30 21:16:56.826953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:56.826923 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:16:56.827445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:56.827110 4526 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:16:58.910783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:58.910742 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:58.913259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:58.913224 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/823598bc-abd8-49f6-91d9-2eb21ec1435c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4z7zj\" (UID: \"823598bc-abd8-49f6-91d9-2eb21ec1435c\") " pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:59.143229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:59.143199 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4z7zj" Oct 30 21:16:59.593967 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:16:59.593935 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4z7zj"] Oct 30 21:16:59.634795 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:16:59.634758 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823598bc_abd8_49f6_91d9_2eb21ec1435c.slice/crio-2a97c6ff7b1fbba59ce906a90874de2cd9b3d1356d90739a0e4ac1d9307b8585 WatchSource:0}: Error finding container 2a97c6ff7b1fbba59ce906a90874de2cd9b3d1356d90739a0e4ac1d9307b8585: Status 404 returned error can't find the container with id 2a97c6ff7b1fbba59ce906a90874de2cd9b3d1356d90739a0e4ac1d9307b8585 Oct 30 21:17:00.066875 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:00.066837 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4z7zj" event={"ID":"823598bc-abd8-49f6-91d9-2eb21ec1435c","Type":"ContainerStarted","Data":"20df68feb63f4be57561f9d323129b2f4bdf089bb9f4fe4f4c77ba9413dc0bb8"} Oct 30 21:17:00.067398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:00.066894 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4z7zj" event={"ID":"823598bc-abd8-49f6-91d9-2eb21ec1435c","Type":"ContainerStarted","Data":"2a97c6ff7b1fbba59ce906a90874de2cd9b3d1356d90739a0e4ac1d9307b8585"} Oct 30 21:17:00.182191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:00.182154 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:17:00.671573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:00.671541 4526 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-26.us-west-1.compute.internal" event="NodeReady" Oct 30 21:17:02.074264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:02.074224 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4z7zj" event={"ID":"823598bc-abd8-49f6-91d9-2eb21ec1435c","Type":"ContainerStarted","Data":"41804a77a00d2abccc0fa358f7f9c1dcc34d6d027be7d719ace53126a49f6fbe"} Oct 30 21:17:02.831558 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:02.831528 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk6xz" Oct 30 21:17:03.240395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.240315 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2"] Oct 30 21:17:03.276882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.276856 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" Oct 30 21:17:03.329062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.329027 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2"] Oct 30 21:17:03.339824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.339781 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-75wfh\"" Oct 30 21:17:03.339824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.339781 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:17:03.344108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.344081 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce6d7395-bbc5-45c8-9dec-f9e89ebdd080-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t4hc2\" (UID: \"ce6d7395-bbc5-45c8-9dec-f9e89ebdd080\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" Oct 30 21:17:03.350443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.350412 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-5kpt2"] Oct 30 21:17:03.371874 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.371839 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f549d567f-ccdx2"] Oct 30 21:17:03.372029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.371999 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:03.385974 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.385951 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.416138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.416106 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5kpt2"] Oct 30 21:17:03.419510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.419465 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f549d567f-ccdx2"] Oct 30 21:17:03.424232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.424207 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:17:03.424414 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.424253 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:17:03.434008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.433985 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-k7m6c\"" Oct 30 21:17:03.444969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.444932 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cdc913d-a5d4-4169-8854-6ef19393ef67-trusted-ca\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.444969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.444961 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5557\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-kube-api-access-q5557\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.445152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.444983 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cdc913d-a5d4-4169-8854-6ef19393ef67-ca-trust-extracted\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.445152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.445042 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cdc913d-a5d4-4169-8854-6ef19393ef67-registry-certificates\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.445152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.445113 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c09590b4-5b5d-4af7-af7b-8c0deffe6481-tmp\") pod \"downloads-6dd789c8b7-5kpt2\" (UID: \"c09590b4-5b5d-4af7-af7b-8c0deffe6481\") " pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:03.445242 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.445194 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-registry-tls\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.445242 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.445220 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cdc913d-a5d4-4169-8854-6ef19393ef67-installation-pull-secrets\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.445302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.445246 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cdc913d-a5d4-4169-8854-6ef19393ef67-image-registry-private-configuration\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.445302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.445295 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce6d7395-bbc5-45c8-9dec-f9e89ebdd080-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t4hc2\" (UID: \"ce6d7395-bbc5-45c8-9dec-f9e89ebdd080\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" Oct 30 21:17:03.445360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.445347 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-bound-sa-token\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.445391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.445365 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkcjh\" (UniqueName: \"kubernetes.io/projected/c09590b4-5b5d-4af7-af7b-8c0deffe6481-kube-api-access-bkcjh\") pod \"downloads-6dd789c8b7-5kpt2\" (UID: \"c09590b4-5b5d-4af7-af7b-8c0deffe6481\") " pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:03.447815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.447799 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce6d7395-bbc5-45c8-9dec-f9e89ebdd080-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t4hc2\" (UID: \"ce6d7395-bbc5-45c8-9dec-f9e89ebdd080\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" Oct 30 21:17:03.487566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.487531 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:17:03.533839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.533781 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:17:03.533970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.533953 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:17:03.534008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.533972 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fxr9c\"" Oct 30 21:17:03.534008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.533994 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:17:03.546174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546152 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-bound-sa-token\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546180 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkcjh\" (UniqueName: \"kubernetes.io/projected/c09590b4-5b5d-4af7-af7b-8c0deffe6481-kube-api-access-bkcjh\") pod \"downloads-6dd789c8b7-5kpt2\" (UID: \"c09590b4-5b5d-4af7-af7b-8c0deffe6481\") " pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:03.546251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546210 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cdc913d-a5d4-4169-8854-6ef19393ef67-trusted-ca\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546231 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5557\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-kube-api-access-q5557\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546259 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cdc913d-a5d4-4169-8854-6ef19393ef67-ca-trust-extracted\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546288 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cdc913d-a5d4-4169-8854-6ef19393ef67-registry-certificates\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546312 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c09590b4-5b5d-4af7-af7b-8c0deffe6481-tmp\") pod \"downloads-6dd789c8b7-5kpt2\" (UID: \"c09590b4-5b5d-4af7-af7b-8c0deffe6481\") " pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:03.546356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546347 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-registry-tls\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546373 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cdc913d-a5d4-4169-8854-6ef19393ef67-installation-pull-secrets\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546402 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cdc913d-a5d4-4169-8854-6ef19393ef67-image-registry-private-configuration\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546750 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cdc913d-a5d4-4169-8854-6ef19393ef67-ca-trust-extracted\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.546832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.546794 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c09590b4-5b5d-4af7-af7b-8c0deffe6481-tmp\") pod \"downloads-6dd789c8b7-5kpt2\" (UID: \"c09590b4-5b5d-4af7-af7b-8c0deffe6481\") " pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:03.547325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.547300 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cdc913d-a5d4-4169-8854-6ef19393ef67-registry-certificates\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.547615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.547599 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cdc913d-a5d4-4169-8854-6ef19393ef67-trusted-ca\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.549084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.549063 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cdc913d-a5d4-4169-8854-6ef19393ef67-installation-pull-secrets\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.549199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.549092 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cdc913d-a5d4-4169-8854-6ef19393ef67-image-registry-private-configuration\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.549260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.549201 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-registry-tls\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.600265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.600237 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" Oct 30 21:17:03.602647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.602626 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-bound-sa-token\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.602870 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.602850 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkcjh\" (UniqueName: \"kubernetes.io/projected/c09590b4-5b5d-4af7-af7b-8c0deffe6481-kube-api-access-bkcjh\") pod \"downloads-6dd789c8b7-5kpt2\" (UID: \"c09590b4-5b5d-4af7-af7b-8c0deffe6481\") " pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:03.602919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.602859 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5557\" (UniqueName: \"kubernetes.io/projected/4cdc913d-a5d4-4169-8854-6ef19393ef67-kube-api-access-q5557\") pod \"image-registry-6f549d567f-ccdx2\" (UID: \"4cdc913d-a5d4-4169-8854-6ef19393ef67\") " pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:03.692335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.692314 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:03.712732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:03.712702 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:04.053095 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:04.053010 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2"] Oct 30 21:17:04.056591 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:17:04.056566 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6d7395_bbc5_45c8_9dec_f9e89ebdd080.slice/crio-f9f78b91d137208f955fa73f7246946c2801c6f7ca5b4462a1b8d1a15989d0f8 WatchSource:0}: Error finding container f9f78b91d137208f955fa73f7246946c2801c6f7ca5b4462a1b8d1a15989d0f8: Status 404 returned error can't find the container with id f9f78b91d137208f955fa73f7246946c2801c6f7ca5b4462a1b8d1a15989d0f8 Oct 30 21:17:04.082858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:04.082824 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4z7zj" event={"ID":"823598bc-abd8-49f6-91d9-2eb21ec1435c","Type":"ContainerStarted","Data":"703be1a8e1f633875137109a0a8c829c4c49383fb0ef3e17c3db68f960e7864b"} Oct 30 21:17:04.084000 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:04.083979 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" event={"ID":"ce6d7395-bbc5-45c8-9dec-f9e89ebdd080","Type":"ContainerStarted","Data":"f9f78b91d137208f955fa73f7246946c2801c6f7ca5b4462a1b8d1a15989d0f8"} Oct 30 21:17:04.104393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:04.104365 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5kpt2"] Oct 30 21:17:04.108573 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:17:04.108542 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09590b4_5b5d_4af7_af7b_8c0deffe6481.slice/crio-0aa6347a6351dc024017dcbf15ec920db5dca23fcae6e4283c249cce97e7b3f9 WatchSource:0}: Error finding container 0aa6347a6351dc024017dcbf15ec920db5dca23fcae6e4283c249cce97e7b3f9: Status 404 returned error can't find the container with id 0aa6347a6351dc024017dcbf15ec920db5dca23fcae6e4283c249cce97e7b3f9 Oct 30 21:17:04.158793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:04.158760 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f549d567f-ccdx2"] Oct 30 21:17:04.162447 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:17:04.162421 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdc913d_a5d4_4169_8854_6ef19393ef67.slice/crio-60384434b4f5ce64391156e5916deacfd8820348c8f58f624c8e611656fe6e8b WatchSource:0}: Error finding container 60384434b4f5ce64391156e5916deacfd8820348c8f58f624c8e611656fe6e8b: Status 404 returned error can't find the container with id 60384434b4f5ce64391156e5916deacfd8820348c8f58f624c8e611656fe6e8b Oct 30 21:17:05.055087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:05.055056 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-chx8w" Oct 30 21:17:05.089222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:05.089181 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" event={"ID":"4cdc913d-a5d4-4169-8854-6ef19393ef67","Type":"ContainerStarted","Data":"6d29c7c9d614c9909116ee4c399c8362e92642ee4e3bb9433cb1129ba8895e43"} Oct 30 21:17:05.089412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:05.089252 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:05.089412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:05.089273 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" event={"ID":"4cdc913d-a5d4-4169-8854-6ef19393ef67","Type":"ContainerStarted","Data":"60384434b4f5ce64391156e5916deacfd8820348c8f58f624c8e611656fe6e8b"} Oct 30 21:17:05.090552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:05.090526 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5kpt2" event={"ID":"c09590b4-5b5d-4af7-af7b-8c0deffe6481","Type":"ContainerStarted","Data":"0aa6347a6351dc024017dcbf15ec920db5dca23fcae6e4283c249cce97e7b3f9"} Oct 30 21:17:05.655420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:05.655388 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tljvj" Oct 30 21:17:07.098465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:07.098434 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" event={"ID":"ce6d7395-bbc5-45c8-9dec-f9e89ebdd080","Type":"ContainerStarted","Data":"1d54a64b70a1b032dd47595129785e6761f852382b1809c7f34776a051cf5075"} Oct 30 21:17:07.098934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:07.098653 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" Oct 30 21:17:07.103445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:07.103427 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" Oct 30 21:17:07.210619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:07.210535 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4hc2" podStartSLOduration=2.22059533 podStartE2EDuration="4.210515996s" totalImagesPullingTime="1.989920666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:07.21032527 +0000 UTC m=+17.343698469" watchObservedRunningTime="2026-10-30 21:17:07.210515996 +0000 UTC m=+17.343889194" Oct 30 21:17:07.211322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:07.211109 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" podStartSLOduration=4.211099872 podStartE2EDuration="4.211099872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:05.283904681 +0000 UTC m=+15.417277880" watchObservedRunningTime="2026-10-30 21:17:07.211099872 +0000 UTC m=+17.344473071" Oct 30 21:17:12.480790 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:17:12.480745 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:17:14.661828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.661793 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-df8lc"] Oct 30 21:17:14.668958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.668922 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.723417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.723330 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:17:14.723417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.723369 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:17:14.723699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.723556 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:17:14.732053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732028 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-wtmp\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.732192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732063 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-root\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.732192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732087 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-sys\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.732192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732153 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-accelerators-collector-config\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.732312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732285 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lkn\" (UniqueName: \"kubernetes.io/projected/43b79470-ae8b-4053-94ce-8526d0c99652-kube-api-access-b4lkn\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.732362 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732330 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.732425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732408 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43b79470-ae8b-4053-94ce-8526d0c99652-metrics-client-ca\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.732526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732482 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-textfile\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.732577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.732555 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-tls\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.772980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.772947 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:17:14.772980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.772947 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:17:14.772980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.772989 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:17:14.773298 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.773052 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qn5sj\"" Oct 30 21:17:14.833803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.833761 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43b79470-ae8b-4053-94ce-8526d0c99652-metrics-client-ca\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.833995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.833815 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-textfile\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.833995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.833850 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-tls\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.833995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.833872 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-wtmp\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.833995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.833932 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-root\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.833995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.833965 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-sys\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.833995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.833993 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-accelerators-collector-config\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.834287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.834034 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lkn\" (UniqueName: \"kubernetes.io/projected/43b79470-ae8b-4053-94ce-8526d0c99652-kube-api-access-b4lkn\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.834287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.834059 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-wtmp\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.834287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.834076 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.834287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.834116 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-sys\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.834287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.834158 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43b79470-ae8b-4053-94ce-8526d0c99652-root\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.834539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.834296 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-textfile\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.834539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.834418 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43b79470-ae8b-4053-94ce-8526d0c99652-metrics-client-ca\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.834683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.834658 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-accelerators-collector-config\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.837010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.836985 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.838996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.838977 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43b79470-ae8b-4053-94ce-8526d0c99652-node-exporter-tls\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.894567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.894535 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lkn\" (UniqueName: \"kubernetes.io/projected/43b79470-ae8b-4053-94ce-8526d0c99652-kube-api-access-b4lkn\") pod \"node-exporter-df8lc\" (UID: \"43b79470-ae8b-4053-94ce-8526d0c99652\") " pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.983655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:14.983570 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-df8lc" Oct 30 21:17:14.995324 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:17:14.995294 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b79470_ae8b_4053_94ce_8526d0c99652.slice/crio-ccc78ec2321f9ee7e3a79d535e0406409400dab6510a5b9e1f40b29e68395adf WatchSource:0}: Error finding container ccc78ec2321f9ee7e3a79d535e0406409400dab6510a5b9e1f40b29e68395adf: Status 404 returned error can't find the container with id ccc78ec2321f9ee7e3a79d535e0406409400dab6510a5b9e1f40b29e68395adf Oct 30 21:17:15.127115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:15.127075 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-df8lc" event={"ID":"43b79470-ae8b-4053-94ce-8526d0c99652","Type":"ContainerStarted","Data":"ccc78ec2321f9ee7e3a79d535e0406409400dab6510a5b9e1f40b29e68395adf"} Oct 30 21:17:17.137101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:17.137061 4526 generic.go:356] "Generic (PLEG): container finished" podID="43b79470-ae8b-4053-94ce-8526d0c99652" containerID="b4e0c8fc91d46564eb84ad93392294f1e72eb43377304d67b566a8cc80d395ee" exitCode=0 Oct 30 21:17:17.137658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:17.137117 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-df8lc" event={"ID":"43b79470-ae8b-4053-94ce-8526d0c99652","Type":"ContainerDied","Data":"b4e0c8fc91d46564eb84ad93392294f1e72eb43377304d67b566a8cc80d395ee"} Oct 30 21:17:18.143980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:18.143947 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-df8lc" event={"ID":"43b79470-ae8b-4053-94ce-8526d0c99652","Type":"ContainerStarted","Data":"e4bf7a8a60a880a935baceb9bc563a718b37f109b470df68fca64a1c4b2450a2"} Oct 30 21:17:18.144481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:18.144007 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-df8lc" event={"ID":"43b79470-ae8b-4053-94ce-8526d0c99652","Type":"ContainerStarted","Data":"92cd3ff7f5417c7b8a53c83f22e44dd0a28d63e95a6d2a18089b8abc5b962d89"} Oct 30 21:17:18.259915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:18.259852 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-df8lc" podStartSLOduration=2.697437574 podStartE2EDuration="4.2598369s" totalImagesPullingTime="1.476952811s" totalInitContainerRuntime="85.446515ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:18.258743597 +0000 UTC m=+28.392116799" watchObservedRunningTime="2026-10-30 21:17:18.2598369 +0000 UTC m=+28.393210098" Oct 30 21:17:20.185581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:20.185549 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-79bgk" Oct 30 21:17:23.744572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.744466 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:17:23.748078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.748050 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.793880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.793842 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:17:23.824332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.824272 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:17:23.824332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.824272 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:17:23.824332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.824299 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9nqg2\"" Oct 30 21:17:23.824332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.824311 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:17:23.842243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.842215 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-frt2j600dv2gt\"" Oct 30 21:17:23.842455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.842338 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:17:23.911338 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911293 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-config\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911351 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911383 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911553 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911588 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911615 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911780 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911812 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n748l\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-kube-api-access-n748l\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911843 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.911965 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911943 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.911987 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-config-out\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912015 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912094 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912167 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912185 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912205 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912250 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912281 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912310 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-web-config\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.912344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.912332 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:23.941559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941521 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:17:23.941748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941585 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:17:23.941748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941603 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:17:23.941748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941617 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:17:23.941748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941685 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:17:23.941748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941694 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:17:23.941748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941687 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:17:23.941748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941741 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:17:23.941748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941751 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:17:23.942111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941746 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:17:23.942111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:23.941888 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:17:24.013651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013557 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013625 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-config-out\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013658 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013693 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013734 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013758 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013791 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013819 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013845 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.013885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013870 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-web-config\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013898 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013926 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-config\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013951 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.013977 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.014003 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.014036 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.014061 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.014114 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.014140 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n748l\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-kube-api-access-n748l\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.014266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.014171 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.015003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.014554 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.015063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.015011 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.015213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.015186 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.016576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.016555 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.018356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.017983 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.019976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.019952 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.020732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.020523 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.021221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.021200 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.021714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.021684 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.022543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.022499 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.024163 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.024098 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.024314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.024229 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-config\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.024387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.024345 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.024820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.024566 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.024820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.024692 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.024820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.024782 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.025121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.025098 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-web-config\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.026803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.026777 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-config-out\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.026972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.026950 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.073548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.073513 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n748l\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-kube-api-access-n748l\") pod \"prometheus-k8s-0\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.169889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.169850 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5kpt2" event={"ID":"c09590b4-5b5d-4af7-af7b-8c0deffe6481","Type":"ContainerStarted","Data":"10a6ad7e0b351981761273e37501ae4eda0cc9baefb0f34051f38d4d9e418bee"} Oct 30 21:17:24.170245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.170156 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:24.190872 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.190837 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-5kpt2" Oct 30 21:17:24.283825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.283409 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-5kpt2" podStartSLOduration=2.105974574 podStartE2EDuration="21.283391284s" totalImagesPullingTime="19.17741671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:24.281864488 +0000 UTC m=+34.415237688" watchObservedRunningTime="2026-10-30 21:17:24.283391284 +0000 UTC m=+34.416764481" Oct 30 21:17:24.367546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.367514 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:24.857865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:24.857832 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:17:24.895816 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:17:24.895777 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561a4802_67a0_4da2_bb6d_027d70da638a.slice/crio-5d3f993765bec21647996b59128e018d065c28fe55bf41bff10acd6eb7bf706e WatchSource:0}: Error finding container 5d3f993765bec21647996b59128e018d065c28fe55bf41bff10acd6eb7bf706e: Status 404 returned error can't find the container with id 5d3f993765bec21647996b59128e018d065c28fe55bf41bff10acd6eb7bf706e Oct 30 21:17:25.176066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:25.175975 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerStarted","Data":"5d3f993765bec21647996b59128e018d065c28fe55bf41bff10acd6eb7bf706e"} Oct 30 21:17:26.100644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:26.100605 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f549d567f-ccdx2" Oct 30 21:17:28.192082 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:28.192038 4526 generic.go:356] "Generic (PLEG): container finished" podID="561a4802-67a0-4da2-bb6d-027d70da638a" containerID="956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24" exitCode=0 Oct 30 21:17:28.192701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:28.192141 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerDied","Data":"956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24"} Oct 30 21:17:33.216142 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:33.216060 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerStarted","Data":"e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c"} Oct 30 21:17:33.216615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:33.216150 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerStarted","Data":"37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791"} Oct 30 21:17:36.229539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:36.229450 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerStarted","Data":"fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90"} Oct 30 21:17:36.229539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:36.229534 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerStarted","Data":"073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2"} Oct 30 21:17:36.230010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:36.229562 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerStarted","Data":"dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff"} Oct 30 21:17:36.230010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:36.229600 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerStarted","Data":"9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871"} Oct 30 21:17:36.370612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:36.370560 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.9639123720000002 podStartE2EDuration="13.370544619s" totalImagesPullingTime="9.38436548s" totalInitContainerRuntime="22.266767ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:36.369061189 +0000 UTC m=+46.502434386" watchObservedRunningTime="2026-10-30 21:17:36.370544619 +0000 UTC m=+46.503917855" Oct 30 21:17:39.368226 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:39.368184 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:17:42.476381 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:17:42.476342 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:17:52.922701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:52.922648 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-cpc25-sxdlt"] Oct 30 21:17:52.923159 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:52.923003 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" podUID="8023590c-1d50-48e6-93e4-683b8368c13c" containerName="insights-gathering" containerID="cri-o://285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771" gracePeriod=30 Oct 30 21:17:53.199822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.199739 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:17:53.254749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.254713 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle\") pod \"8023590c-1d50-48e6-93e4-683b8368c13c\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " Oct 30 21:17:53.254926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.254811 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path\") pod \"8023590c-1d50-48e6-93e4-683b8368c13c\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " Oct 30 21:17:53.254926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.254829 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle\") pod \"8023590c-1d50-48e6-93e4-683b8368c13c\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " Oct 30 21:17:53.254926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.254863 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj\") pod \"8023590c-1d50-48e6-93e4-683b8368c13c\" (UID: \"8023590c-1d50-48e6-93e4-683b8368c13c\") " Oct 30 21:17:53.255132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.255107 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle" pod "8023590c-1d50-48e6-93e4-683b8368c13c" (UID: "8023590c-1d50-48e6-93e4-683b8368c13c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:53.255192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.255170 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle" pod "8023590c-1d50-48e6-93e4-683b8368c13c" (UID: "8023590c-1d50-48e6-93e4-683b8368c13c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:53.255192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.255180 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path" pod "8023590c-1d50-48e6-93e4-683b8368c13c" (UID: "8023590c-1d50-48e6-93e4-683b8368c13c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:53.257319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.257298 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj" pod "8023590c-1d50-48e6-93e4-683b8368c13c" (UID: "8023590c-1d50-48e6-93e4-683b8368c13c"). InnerVolumeSpecName "kube-api-access-nvfmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:53.291512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.291466 4526 generic.go:356] "Generic (PLEG): container finished" podID="8023590c-1d50-48e6-93e4-683b8368c13c" containerID="285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771" exitCode=2 Oct 30 21:17:53.291693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.291531 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" event={"ID":"8023590c-1d50-48e6-93e4-683b8368c13c","Type":"ContainerDied","Data":"285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771"} Oct 30 21:17:53.291693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.291554 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" Oct 30 21:17:53.291693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.291567 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cpc25-sxdlt" event={"ID":"8023590c-1d50-48e6-93e4-683b8368c13c","Type":"ContainerDied","Data":"cc60dfa3b6d1921fa964050e1fdde4e3377d80e99643e5f43eed09ae6a6e4b5c"} Oct 30 21:17:53.291693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.291586 4526 scope.go:122] "RemoveContainer" containerID="285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771" Oct 30 21:17:53.300026 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.300003 4526 scope.go:122] "RemoveContainer" containerID="285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771" Oct 30 21:17:53.300340 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:17:53.300317 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771\": container with ID starting with 285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771 not found: ID does not exist" containerID="285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771" Oct 30 21:17:53.300395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.300351 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771"} err="failed to get container status \"285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771\": rpc error: code = NotFound desc = could not find container \"285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771\": container with ID starting with 285ecdee142d7e0320e7a9db317ce7de43a2a1bd6e83d0b5461877c13cce3771 not found: ID does not exist" Oct 30 21:17:53.355641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.355595 4526 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8023590c-1d50-48e6-93e4-683b8368c13c-archives-path\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:53.355641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.355639 4526 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-trusted-ca-bundle\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:53.355641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.355649 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/8023590c-1d50-48e6-93e4-683b8368c13c-kube-api-access-nvfmj\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:53.355876 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.355660 4526 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8023590c-1d50-48e6-93e4-683b8368c13c-service-ca-bundle\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:53.420167 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.420128 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-cpc25-sxdlt"] Oct 30 21:17:53.424210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.424182 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-cpc25-sxdlt"] Oct 30 21:17:53.982498 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:17:53.982451 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8023590c-1d50-48e6-93e4-683b8368c13c" path="/var/lib/kubelet/pods/8023590c-1d50-48e6-93e4-683b8368c13c/volumes" Oct 30 21:18:12.476107 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:12.476068 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:18:16.799170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:16.799130 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:16.817833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:16.817797 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:17.411263 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:17.411237 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:36.744548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:36.744514 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:18:36.745115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:36.745081 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy" containerID="cri-o://073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2" gracePeriod=600 Oct 30 21:18:36.745206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:36.745145 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy-web" containerID="cri-o://dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff" gracePeriod=600 Oct 30 21:18:36.745262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:36.745175 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="thanos-sidecar" containerID="cri-o://9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871" gracePeriod=600 Oct 30 21:18:36.745262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:36.745194 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="config-reloader" containerID="cri-o://e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c" gracePeriod=600 Oct 30 21:18:36.745366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:36.745166 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90" gracePeriod=600 Oct 30 21:18:36.745366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:36.745072 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="prometheus" containerID="cri-o://37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791" gracePeriod=600 Oct 30 21:18:37.395278 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:37.395218 4526 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 21:18:37.395516 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:37.395298 4526 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 21:18:37.395516 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:37.395325 4526 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 21:18:37.395516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.395345 4526 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 30 21:18:37.465189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.465154 4526 generic.go:356] "Generic (PLEG): container finished" podID="561a4802-67a0-4da2-bb6d-027d70da638a" containerID="fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90" exitCode=0 Oct 30 21:18:37.465189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.465186 4526 generic.go:356] "Generic (PLEG): container finished" podID="561a4802-67a0-4da2-bb6d-027d70da638a" containerID="dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff" exitCode=0 Oct 30 21:18:37.465189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.465195 4526 generic.go:356] "Generic (PLEG): container finished" podID="561a4802-67a0-4da2-bb6d-027d70da638a" containerID="9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871" exitCode=0 Oct 30 21:18:37.465189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.465203 4526 generic.go:356] "Generic (PLEG): container finished" podID="561a4802-67a0-4da2-bb6d-027d70da638a" containerID="e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c" exitCode=0 Oct 30 21:18:37.465480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.465226 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerDied","Data":"fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90"} Oct 30 21:18:37.465480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.465292 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerDied","Data":"dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff"} Oct 30 21:18:37.465480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.465320 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerDied","Data":"9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871"} Oct 30 21:18:37.465480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:37.465344 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerDied","Data":"e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c"} Oct 30 21:18:38.472714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:38.472683 4526 generic.go:356] "Generic (PLEG): container finished" podID="561a4802-67a0-4da2-bb6d-027d70da638a" containerID="073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2" exitCode=0 Oct 30 21:18:38.473180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:38.472728 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerDied","Data":"073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2"} Oct 30 21:18:41.903726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:41.903705 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.033978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.033893 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-1\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.033978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.033949 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-serving-certs-ca-bundle\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.033978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.033968 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-2\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.034268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.033987 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-web-config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-web-config\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.034268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034009 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-config-out\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.034268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034036 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-kube-api-access-n748l\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-kube-api-access-n748l\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.034268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034054 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.034268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034077 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-tls\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.034268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034118 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-metrics-client-ca\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.034603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034346 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-1" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:42.034603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034430 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-serving-certs-ca-bundle" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:42.034603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034467 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-metrics-client-ca" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:42.034603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034563 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-tls-assets\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-tls-assets\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.035231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034943 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-2" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:42.035231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.034961 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-kube-rbac-proxy\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.035481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.035168 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-config\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.035993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.035951 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-kubelet-serving-ca-bundle" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:42.036162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.036107 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-kubelet-serving-ca-bundle\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.036591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.036285 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-thanos-prometheus-http-client-file\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.037176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.036729 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.037339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037311 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-grpc-tls\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.037454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037020 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.037454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037362 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-db\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.037454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037396 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-metrics-client-certs\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.037454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037404 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-tls" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.037454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037426 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-0\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.037760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037457 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-trusted-ca-bundle\") pod \"561a4802-67a0-4da2-bb6d-027d70da638a\" (UID: \"561a4802-67a0-4da2-bb6d-027d70da638a\") " Oct 30 21:18:42.037856 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037836 4526 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.037978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.037963 4526 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.038075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.038062 4526 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.038171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.038157 4526 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.038259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.038247 4526 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.038347 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.038335 4526 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-metrics-client-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.038432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.038419 4526 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.038528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.038102 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-kube-api-access-n748l" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "kube-api-access-n748l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:42.038635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.038432 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-trusted-ca-bundle" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:42.038635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.038582 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-config-out" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:42.039065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.039012 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-kube-rbac-proxy" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.039418 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.039366 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-db" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:42.039751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.039699 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-0" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:42.039968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.039916 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-config" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.039968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.039930 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-tls-assets" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:42.039968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.039940 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-grpc-tls" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.040200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.040176 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-thanos-prometheus-http-client-file" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.040625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.040597 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-thanos-sidecar-tls" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.040818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.040800 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-metrics-client-certs" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.048887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.048863 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-web-config" pod "561a4802-67a0-4da2-bb6d-027d70da638a" (UID: "561a4802-67a0-4da2-bb6d-027d70da638a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:42.139403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139369 4526 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-tls-assets\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139397 4526 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-kube-rbac-proxy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139406 4526 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-config\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139416 4526 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139427 4526 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139436 4526 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-grpc-tls\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139444 4526 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-db\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139453 4526 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-secret-metrics-client-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139461 4526 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139470 4526 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/561a4802-67a0-4da2-bb6d-027d70da638a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139523 4526 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/561a4802-67a0-4da2-bb6d-027d70da638a-web-config\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139534 4526 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/561a4802-67a0-4da2-bb6d-027d70da638a-config-out\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.139660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.139543 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n748l\" (UniqueName: \"kubernetes.io/projected/561a4802-67a0-4da2-bb6d-027d70da638a-kube-api-access-n748l\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:42.475690 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:42.475610 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:18:42.488471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.488442 4526 generic.go:356] "Generic (PLEG): container finished" podID="561a4802-67a0-4da2-bb6d-027d70da638a" containerID="37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791" exitCode=0 Oct 30 21:18:42.488660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.488527 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerDied","Data":"37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791"} Oct 30 21:18:42.488660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.488545 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.488660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.488595 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"561a4802-67a0-4da2-bb6d-027d70da638a","Type":"ContainerDied","Data":"5d3f993765bec21647996b59128e018d065c28fe55bf41bff10acd6eb7bf706e"} Oct 30 21:18:42.488660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.488625 4526 scope.go:122] "RemoveContainer" containerID="fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90" Oct 30 21:18:42.496201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.496180 4526 scope.go:122] "RemoveContainer" containerID="073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2" Oct 30 21:18:42.502883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.502862 4526 scope.go:122] "RemoveContainer" containerID="dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff" Oct 30 21:18:42.509271 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.509257 4526 scope.go:122] "RemoveContainer" containerID="9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871" Oct 30 21:18:42.515656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.515638 4526 scope.go:122] "RemoveContainer" containerID="e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c" Oct 30 21:18:42.521897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.521879 4526 scope.go:122] "RemoveContainer" containerID="37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791" Oct 30 21:18:42.528577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.528549 4526 scope.go:122] "RemoveContainer" containerID="956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24" Oct 30 21:18:42.535043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.535026 4526 scope.go:122] "RemoveContainer" containerID="fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90" Oct 30 21:18:42.535281 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:42.535266 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90\": container with ID starting with fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90 not found: ID does not exist" containerID="fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90" Oct 30 21:18:42.535325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.535292 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90"} err="failed to get container status \"fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90\": rpc error: code = NotFound desc = could not find container \"fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90\": container with ID starting with fc491873548efa008ec192f46a293647b59046b9202cda8f65ce24ae12ebab90 not found: ID does not exist" Oct 30 21:18:42.535325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.535312 4526 scope.go:122] "RemoveContainer" containerID="073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2" Oct 30 21:18:42.535516 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:42.535500 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2\": container with ID starting with 073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2 not found: ID does not exist" containerID="073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2" Oct 30 21:18:42.535560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.535521 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2"} err="failed to get container status \"073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2\": rpc error: code = NotFound desc = could not find container \"073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2\": container with ID starting with 073ea7f1bbf96a56650300554f9a864314c24da5b6586bbe8afee4175242cde2 not found: ID does not exist" Oct 30 21:18:42.535560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.535534 4526 scope.go:122] "RemoveContainer" containerID="dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff" Oct 30 21:18:42.535753 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:42.535734 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff\": container with ID starting with dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff not found: ID does not exist" containerID="dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff" Oct 30 21:18:42.535793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.535759 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff"} err="failed to get container status \"dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff\": rpc error: code = NotFound desc = could not find container \"dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff\": container with ID starting with dbfb190f7bcbaeec2f8921e34451903bc825880bad53c0cfc90b2c2024c112ff not found: ID does not exist" Oct 30 21:18:42.535793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.535774 4526 scope.go:122] "RemoveContainer" containerID="9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871" Oct 30 21:18:42.536038 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:42.536023 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871\": container with ID starting with 9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871 not found: ID does not exist" containerID="9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871" Oct 30 21:18:42.536102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.536040 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871"} err="failed to get container status \"9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871\": rpc error: code = NotFound desc = could not find container \"9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871\": container with ID starting with 9c9f1ef8a6d7f5d5a157d7de525622b0cad3193b1af18c100be76121bc366871 not found: ID does not exist" Oct 30 21:18:42.536102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.536051 4526 scope.go:122] "RemoveContainer" containerID="e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c" Oct 30 21:18:42.536286 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:42.536272 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c\": container with ID starting with e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c not found: ID does not exist" containerID="e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c" Oct 30 21:18:42.536326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.536289 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c"} err="failed to get container status \"e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c\": rpc error: code = NotFound desc = could not find container \"e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c\": container with ID starting with e68c9a97c4db4c27f151a2da0bc75b52e2a7a3d7e0414feed7f2b8fd3bfa120c not found: ID does not exist" Oct 30 21:18:42.536326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.536301 4526 scope.go:122] "RemoveContainer" containerID="37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791" Oct 30 21:18:42.536534 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:42.536513 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791\": container with ID starting with 37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791 not found: ID does not exist" containerID="37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791" Oct 30 21:18:42.536591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.536540 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791"} err="failed to get container status \"37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791\": rpc error: code = NotFound desc = could not find container \"37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791\": container with ID starting with 37864b4e1f35a860f1981c38a19968fd194364d9b234dfebdfdb5ffbea9bd791 not found: ID does not exist" Oct 30 21:18:42.536591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.536579 4526 scope.go:122] "RemoveContainer" containerID="956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24" Oct 30 21:18:42.536845 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:18:42.536829 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24\": container with ID starting with 956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24 not found: ID does not exist" containerID="956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24" Oct 30 21:18:42.536890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.536851 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24"} err="failed to get container status \"956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24\": rpc error: code = NotFound desc = could not find container \"956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24\": container with ID starting with 956b9513acb3de312663238e7b83c3d398c0536290e6e231f854f472e98b9f24 not found: ID does not exist" Oct 30 21:18:42.655706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.655678 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:18:42.705997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.705805 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:18:42.707212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.706856 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:18:42.707212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707186 4526 cpu_manager.go:414] "containerMap: removing container" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy-web" Oct 30 21:18:42.707212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707201 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy-web" Oct 30 21:18:42.707212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707219 4526 cpu_manager.go:414] "containerMap: removing container" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="config-reloader" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707225 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="config-reloader" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707232 4526 cpu_manager.go:414] "containerMap: removing container" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707239 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707263 4526 cpu_manager.go:414] "containerMap: removing container" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="thanos-sidecar" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707268 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="thanos-sidecar" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707279 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8023590c-1d50-48e6-93e4-683b8368c13c" containerName="insights-gathering" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707287 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8023590c-1d50-48e6-93e4-683b8368c13c" containerName="insights-gathering" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707298 4526 cpu_manager.go:414] "containerMap: removing container" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy-thanos" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707304 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy-thanos" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707312 4526 cpu_manager.go:414] "containerMap: removing container" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="prometheus" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707317 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="prometheus" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707324 4526 cpu_manager.go:414] "containerMap: removing container" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="init-config-reloader" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707329 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="init-config-reloader" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707373 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="thanos-sidecar" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707382 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8023590c-1d50-48e6-93e4-683b8368c13c" containerName="insights-gathering" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707387 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy-thanos" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707392 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="prometheus" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707398 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy-web" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707404 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="config-reloader" Oct 30 21:18:42.707447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.707409 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" containerName="kube-rbac-proxy" Oct 30 21:18:42.711291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.711274 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.763972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.763879 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:18:42.763972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.763954 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:18:42.764166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.763980 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:18:42.764166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.763997 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:18:42.764625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.764605 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:18:42.813402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.813372 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:18:42.813402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.813393 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:18:42.813402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.813375 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:18:42.813707 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.813630 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:18:42.813986 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.813972 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:18:42.814076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.814057 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:18:42.814231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.814218 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:18:42.814407 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.814394 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:18:42.814477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.814467 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-frt2j600dv2gt\"" Oct 30 21:18:42.814550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.814517 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:18:42.814550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.814528 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:18:42.814659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.814568 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9nqg2\"" Oct 30 21:18:42.830767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.830746 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:18:42.845446 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845422 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845453 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845475 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5j5\" (UniqueName: \"kubernetes.io/projected/9b8783f8-7c91-419f-820c-e75619148153-kube-api-access-ct5j5\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845600 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845669 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845633 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845679 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845699 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845720 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b8783f8-7c91-419f-820c-e75619148153-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845736 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845779 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845808 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845868 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845902 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845949 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845918 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.845949 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845944 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.846008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.845963 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.846040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.846022 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-config\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.846073 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.846041 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.846105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.846086 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b8783f8-7c91-419f-820c-e75619148153-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.846137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.846106 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.946801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.946768 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.946801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.946801 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.946824 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5j5\" (UniqueName: \"kubernetes.io/projected/9b8783f8-7c91-419f-820c-e75619148153-kube-api-access-ct5j5\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.946843 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.946866 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.946895 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.946975 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947013 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b8783f8-7c91-419f-820c-e75619148153-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947038 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947065 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947089 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947121 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947199 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947231 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947276 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947310 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.947343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947351 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-config\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.948178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947378 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.948178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947412 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b8783f8-7c91-419f-820c-e75619148153-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.948178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947440 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.948178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947636 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.948178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.947821 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.948178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.948097 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.950628 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.949305 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.950628 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.949618 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.950628 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.949976 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.950628 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.950192 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.950628 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.950591 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.950949 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.950851 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-config\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.951265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.951243 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b8783f8-7c91-419f-820c-e75619148153-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.951663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.951639 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.952267 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.952037 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.952267 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.952207 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.952267 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.952209 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b8783f8-7c91-419f-820c-e75619148153-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.952966 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.952900 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b8783f8-7c91-419f-820c-e75619148153-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.953055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.952963 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.953273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.953255 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.953403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.953385 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:42.953682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:42.953667 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b8783f8-7c91-419f-820c-e75619148153-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:43.004349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:43.004320 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5j5\" (UniqueName: \"kubernetes.io/projected/9b8783f8-7c91-419f-820c-e75619148153-kube-api-access-ct5j5\") pod \"prometheus-k8s-0\" (UID: \"9b8783f8-7c91-419f-820c-e75619148153\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:43.028410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:43.028334 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:18:43.456591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:43.456553 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:18:43.460088 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:18:43.460061 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8783f8_7c91_419f_820c_e75619148153.slice/crio-fb3f43d18fdf17751d43decbebe5f92fa5d19b9e3944cb1a2f08bffb831ef033 WatchSource:0}: Error finding container fb3f43d18fdf17751d43decbebe5f92fa5d19b9e3944cb1a2f08bffb831ef033: Status 404 returned error can't find the container with id fb3f43d18fdf17751d43decbebe5f92fa5d19b9e3944cb1a2f08bffb831ef033 Oct 30 21:18:43.496449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:43.496422 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b8783f8-7c91-419f-820c-e75619148153","Type":"ContainerStarted","Data":"fb3f43d18fdf17751d43decbebe5f92fa5d19b9e3944cb1a2f08bffb831ef033"} Oct 30 21:18:43.982591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:43.982553 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561a4802-67a0-4da2-bb6d-027d70da638a" path="/var/lib/kubelet/pods/561a4802-67a0-4da2-bb6d-027d70da638a/volumes" Oct 30 21:18:44.501777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:44.501747 4526 generic.go:356] "Generic (PLEG): container finished" podID="9b8783f8-7c91-419f-820c-e75619148153" containerID="0e6ce07979e1ad83124c65621cc2ffc9b3892763397c61d24952d4093fd8c565" exitCode=0 Oct 30 21:18:44.501935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:44.501823 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b8783f8-7c91-419f-820c-e75619148153","Type":"ContainerDied","Data":"0e6ce07979e1ad83124c65621cc2ffc9b3892763397c61d24952d4093fd8c565"} Oct 30 21:18:45.508460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:45.508423 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b8783f8-7c91-419f-820c-e75619148153","Type":"ContainerStarted","Data":"70a28854e3bbbf8175c734dc13a86465f14c9a67aa781b06824a09280e3c9ee9"} Oct 30 21:18:45.508897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:45.508515 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b8783f8-7c91-419f-820c-e75619148153","Type":"ContainerStarted","Data":"dc92d3588ac03c03b8105be2f13e043690c7c681190c4ea78802ad2bb1247230"} Oct 30 21:18:45.508897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:45.508543 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b8783f8-7c91-419f-820c-e75619148153","Type":"ContainerStarted","Data":"6e4eb6cff194c3178a4564af511caec08fe650169d482a360c3cd21bb6d640f3"} Oct 30 21:18:45.508897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:45.508568 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b8783f8-7c91-419f-820c-e75619148153","Type":"ContainerStarted","Data":"19b49bbf42f8fda1a3cc30e6e4d1d45210c19f1f02631ae1ccdc879102b222de"} Oct 30 21:18:45.508897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:45.508594 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b8783f8-7c91-419f-820c-e75619148153","Type":"ContainerStarted","Data":"cbfdf0572459af695cccf7be86fd04ecb3228f747b6eeb0cfdb7b8ddf755825c"} Oct 30 21:18:45.508897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:45.508618 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b8783f8-7c91-419f-820c-e75619148153","Type":"ContainerStarted","Data":"450045dedb7cf9fc2d9e227b78d67fee78e6e80a63cee55d1489831a133894a0"} Oct 30 21:18:45.634840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:45.634777 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.616396623 podStartE2EDuration="3.634761381s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.364758ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:45.633256938 +0000 UTC m=+115.766630170" watchObservedRunningTime="2026-10-30 21:18:45.634761381 +0000 UTC m=+115.768134579" Oct 30 21:18:48.029329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:18:48.029287 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:19:12.476173 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:19:12.476086 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:19:42.476515 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:19:42.476459 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:19:43.029521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:19:43.029465 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:19:43.045791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:19:43.045766 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:19:43.712887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:19:43.712862 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:20:12.476717 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:20:12.476672 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:20:42.476668 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:20:42.476583 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:21:12.476944 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:21:12.476903 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:21:34.362808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.362776 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv"] Oct 30 21:21:34.364799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.364783 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.427193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.427166 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:21:34.427750 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.427732 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:21:34.427828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.427811 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:21:34.477070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.477046 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:21:34.477070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.477046 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:21:34.477275 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.477125 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-929kb\"" Oct 30 21:21:34.478512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.478477 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv"] Oct 30 21:21:34.513880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.513845 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzl5h\" (UniqueName: \"kubernetes.io/projected/e55f8e09-8472-4097-bbfa-3ae7edb3525b-kube-api-access-fzl5h\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.514052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.513887 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e55f8e09-8472-4097-bbfa-3ae7edb3525b-tmp\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.514052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.513917 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e55f8e09-8472-4097-bbfa-3ae7edb3525b-manifests\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.514052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.513978 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.514052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.514027 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-webhook-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.514201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.514118 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-metrics-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.615265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.615182 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzl5h\" (UniqueName: \"kubernetes.io/projected/e55f8e09-8472-4097-bbfa-3ae7edb3525b-kube-api-access-fzl5h\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.615265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.615231 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e55f8e09-8472-4097-bbfa-3ae7edb3525b-tmp\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.615265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.615259 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e55f8e09-8472-4097-bbfa-3ae7edb3525b-manifests\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.615564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.615273 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.615564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.615289 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-webhook-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.615564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.615318 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-metrics-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.615747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.615716 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e55f8e09-8472-4097-bbfa-3ae7edb3525b-manifests\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.617887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.617864 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e55f8e09-8472-4097-bbfa-3ae7edb3525b-tmp\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.618011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.617993 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-webhook-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.618216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.618193 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.618275 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.618198 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e55f8e09-8472-4097-bbfa-3ae7edb3525b-metrics-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.673026 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.672993 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzl5h\" (UniqueName: \"kubernetes.io/projected/e55f8e09-8472-4097-bbfa-3ae7edb3525b-kube-api-access-fzl5h\") pod \"opendatahub-operator-controller-manager-85bb549b7d-nr8tv\" (UID: \"e55f8e09-8472-4097-bbfa-3ae7edb3525b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:34.676829 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:34.676808 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:35.160939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:35.160776 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv"] Oct 30 21:21:35.163549 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:21:35.163522 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55f8e09_8472_4097_bbfa_3ae7edb3525b.slice/crio-a9768bb160ce6707348212aa64488102a6d87f9a68fc50eea3dce9cbd5a08695 WatchSource:0}: Error finding container a9768bb160ce6707348212aa64488102a6d87f9a68fc50eea3dce9cbd5a08695: Status 404 returned error can't find the container with id a9768bb160ce6707348212aa64488102a6d87f9a68fc50eea3dce9cbd5a08695 Oct 30 21:21:36.061546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:36.061510 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" event={"ID":"e55f8e09-8472-4097-bbfa-3ae7edb3525b","Type":"ContainerStarted","Data":"a9768bb160ce6707348212aa64488102a6d87f9a68fc50eea3dce9cbd5a08695"} Oct 30 21:21:39.074566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:39.074529 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" event={"ID":"e55f8e09-8472-4097-bbfa-3ae7edb3525b","Type":"ContainerStarted","Data":"1d44f436f857bf80954edc81b9227d93005083de33dce082672883550ae3af3d"} Oct 30 21:21:40.079769 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:40.079733 4526 generic.go:356] "Generic (PLEG): container finished" podID="e55f8e09-8472-4097-bbfa-3ae7edb3525b" containerID="1d44f436f857bf80954edc81b9227d93005083de33dce082672883550ae3af3d" exitCode=0 Oct 30 21:21:40.080259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:40.079782 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" event={"ID":"e55f8e09-8472-4097-bbfa-3ae7edb3525b","Type":"ContainerDied","Data":"1d44f436f857bf80954edc81b9227d93005083de33dce082672883550ae3af3d"} Oct 30 21:21:41.086717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:41.086679 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" event={"ID":"e55f8e09-8472-4097-bbfa-3ae7edb3525b","Type":"ContainerStarted","Data":"fe0d000e0c220fd92285334aff081b41abfc7c5eb82f3d97bfe7dcd1ea3a585a"} Oct 30 21:21:41.087173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:41.086796 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:41.314227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:41.314160 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" podStartSLOduration=2.539457853 podStartE2EDuration="7.314140238s" totalImagesPullingTime="4.315684591s" totalInitContainerRuntime="458.997794ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:41.264693426 +0000 UTC m=+291.398066625" watchObservedRunningTime="2026-10-30 21:21:41.314140238 +0000 UTC m=+291.447513438" Oct 30 21:21:42.476352 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:21:42.476309 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:21:52.092281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:52.092245 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-nr8tv" Oct 30 21:21:56.305248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.305215 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf"] Oct 30 21:21:56.312204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.312183 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" Oct 30 21:21:56.366400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.366371 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:21:56.366612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.366583 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:21:56.388134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.388101 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-457dp\" (UniqueName: \"kubernetes.io/projected/7791baec-e4c0-4437-bb69-e86aec286095-kube-api-access-457dp\") pod \"cert-manager-cainjector-5c57bbd4fc-9dvcf\" (UID: \"7791baec-e4c0-4437-bb69-e86aec286095\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" Oct 30 21:21:56.388309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.388174 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7791baec-e4c0-4437-bb69-e86aec286095-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9dvcf\" (UID: \"7791baec-e4c0-4437-bb69-e86aec286095\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" Oct 30 21:21:56.416085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.416062 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hdcn8\"" Oct 30 21:21:56.416993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.416972 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf"] Oct 30 21:21:56.489393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.489364 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7791baec-e4c0-4437-bb69-e86aec286095-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9dvcf\" (UID: \"7791baec-e4c0-4437-bb69-e86aec286095\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" Oct 30 21:21:56.489588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.489410 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-457dp\" (UniqueName: \"kubernetes.io/projected/7791baec-e4c0-4437-bb69-e86aec286095-kube-api-access-457dp\") pod \"cert-manager-cainjector-5c57bbd4fc-9dvcf\" (UID: \"7791baec-e4c0-4437-bb69-e86aec286095\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" Oct 30 21:21:56.548127 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.548094 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7791baec-e4c0-4437-bb69-e86aec286095-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9dvcf\" (UID: \"7791baec-e4c0-4437-bb69-e86aec286095\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" Oct 30 21:21:56.548255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.548213 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-457dp\" (UniqueName: \"kubernetes.io/projected/7791baec-e4c0-4437-bb69-e86aec286095-kube-api-access-457dp\") pod \"cert-manager-cainjector-5c57bbd4fc-9dvcf\" (UID: \"7791baec-e4c0-4437-bb69-e86aec286095\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" Oct 30 21:21:56.632423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:56.632388 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" Oct 30 21:21:57.007246 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:57.007158 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf"] Oct 30 21:21:57.011234 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:21:57.011208 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7791baec_e4c0_4437_bb69_e86aec286095.slice/crio-e09c4e08aa68a3c39ded3fd4fd1295f17a96cea36afe62aa8487cc064a2c9155 WatchSource:0}: Error finding container e09c4e08aa68a3c39ded3fd4fd1295f17a96cea36afe62aa8487cc064a2c9155: Status 404 returned error can't find the container with id e09c4e08aa68a3c39ded3fd4fd1295f17a96cea36afe62aa8487cc064a2c9155 Oct 30 21:21:57.013221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:57.013203 4526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:21:57.138840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:57.138805 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" event={"ID":"7791baec-e4c0-4437-bb69-e86aec286095","Type":"ContainerStarted","Data":"e09c4e08aa68a3c39ded3fd4fd1295f17a96cea36afe62aa8487cc064a2c9155"} Oct 30 21:21:59.773140 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.771874 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g"] Oct 30 21:21:59.776175 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.776152 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:21:59.820501 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.820455 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdf20d6e-28cb-40ef-ba2b-b2108efbe6da-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vns6g\" (UID: \"cdf20d6e-28cb-40ef-ba2b-b2108efbe6da\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:21:59.820651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.820626 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hj7\" (UniqueName: \"kubernetes.io/projected/cdf20d6e-28cb-40ef-ba2b-b2108efbe6da-kube-api-access-p8hj7\") pod \"cert-manager-webhook-5dbbbf75cc-vns6g\" (UID: \"cdf20d6e-28cb-40ef-ba2b-b2108efbe6da\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:21:59.835278 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.835252 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g"] Oct 30 21:21:59.880508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.880462 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-z5b4h\"" Oct 30 21:21:59.922022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.921980 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hj7\" (UniqueName: \"kubernetes.io/projected/cdf20d6e-28cb-40ef-ba2b-b2108efbe6da-kube-api-access-p8hj7\") pod \"cert-manager-webhook-5dbbbf75cc-vns6g\" (UID: \"cdf20d6e-28cb-40ef-ba2b-b2108efbe6da\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:21:59.922179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.922054 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdf20d6e-28cb-40ef-ba2b-b2108efbe6da-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vns6g\" (UID: \"cdf20d6e-28cb-40ef-ba2b-b2108efbe6da\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:21:59.988926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.988891 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdf20d6e-28cb-40ef-ba2b-b2108efbe6da-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vns6g\" (UID: \"cdf20d6e-28cb-40ef-ba2b-b2108efbe6da\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:21:59.989242 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:21:59.989222 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hj7\" (UniqueName: \"kubernetes.io/projected/cdf20d6e-28cb-40ef-ba2b-b2108efbe6da-kube-api-access-p8hj7\") pod \"cert-manager-webhook-5dbbbf75cc-vns6g\" (UID: \"cdf20d6e-28cb-40ef-ba2b-b2108efbe6da\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:22:00.091597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:00.091559 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:22:00.467624 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:00.467548 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g"] Oct 30 21:22:00.470898 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:22:00.470851 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdf20d6e_28cb_40ef_ba2b_b2108efbe6da.slice/crio-5cca875d385751cb269a24eeade1d87f9feeac0bcb220d5dc9682e3aa0ab6501 WatchSource:0}: Error finding container 5cca875d385751cb269a24eeade1d87f9feeac0bcb220d5dc9682e3aa0ab6501: Status 404 returned error can't find the container with id 5cca875d385751cb269a24eeade1d87f9feeac0bcb220d5dc9682e3aa0ab6501 Oct 30 21:22:01.161964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:01.161878 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" event={"ID":"cdf20d6e-28cb-40ef-ba2b-b2108efbe6da","Type":"ContainerStarted","Data":"5cca875d385751cb269a24eeade1d87f9feeac0bcb220d5dc9682e3aa0ab6501"} Oct 30 21:22:02.166858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:02.166825 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" event={"ID":"7791baec-e4c0-4437-bb69-e86aec286095","Type":"ContainerStarted","Data":"078b236231ecd3a99ff7798e4100ae28595aecf2a5fe4ef2bf84a7e35bab0966"} Oct 30 21:22:02.168458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:02.168436 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" event={"ID":"cdf20d6e-28cb-40ef-ba2b-b2108efbe6da","Type":"ContainerStarted","Data":"419a72b0214281574ecbe2f87756da4bb1163f83a6a1ad608d232780c1040e98"} Oct 30 21:22:02.168594 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:02.168552 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:22:02.282184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:02.282071 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9dvcf" podStartSLOduration=1.959154327 podStartE2EDuration="6.282052973s" totalImagesPullingTime="4.322898646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:02.280441261 +0000 UTC m=+312.413814459" watchObservedRunningTime="2026-10-30 21:22:02.282052973 +0000 UTC m=+312.415426173" Oct 30 21:22:02.395219 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:02.395166 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" podStartSLOduration=1.878612322 podStartE2EDuration="3.395153736s" totalImagesPullingTime="1.516541414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:02.395066425 +0000 UTC m=+312.528439620" watchObservedRunningTime="2026-10-30 21:22:02.395153736 +0000 UTC m=+312.528526933" Oct 30 21:22:03.530271 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.530238 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-l9fr6"] Oct 30 21:22:03.533398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.533375 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-l9fr6" Oct 30 21:22:03.634480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.634441 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gkkk7\"" Oct 30 21:22:03.635584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.635560 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-l9fr6"] Oct 30 21:22:03.657092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.657062 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hkq\" (UniqueName: \"kubernetes.io/projected/38e41b2b-5357-4e1b-8db3-837b19953be0-kube-api-access-n8hkq\") pod \"cert-manager-597b95cb84-l9fr6\" (UID: \"38e41b2b-5357-4e1b-8db3-837b19953be0\") " pod="cert-manager/cert-manager-597b95cb84-l9fr6" Oct 30 21:22:03.657270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.657114 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38e41b2b-5357-4e1b-8db3-837b19953be0-bound-sa-token\") pod \"cert-manager-597b95cb84-l9fr6\" (UID: \"38e41b2b-5357-4e1b-8db3-837b19953be0\") " pod="cert-manager/cert-manager-597b95cb84-l9fr6" Oct 30 21:22:03.758155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.758111 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38e41b2b-5357-4e1b-8db3-837b19953be0-bound-sa-token\") pod \"cert-manager-597b95cb84-l9fr6\" (UID: \"38e41b2b-5357-4e1b-8db3-837b19953be0\") " pod="cert-manager/cert-manager-597b95cb84-l9fr6" Oct 30 21:22:03.758343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.758189 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hkq\" (UniqueName: \"kubernetes.io/projected/38e41b2b-5357-4e1b-8db3-837b19953be0-kube-api-access-n8hkq\") pod \"cert-manager-597b95cb84-l9fr6\" (UID: \"38e41b2b-5357-4e1b-8db3-837b19953be0\") " pod="cert-manager/cert-manager-597b95cb84-l9fr6" Oct 30 21:22:03.816332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.816301 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hkq\" (UniqueName: \"kubernetes.io/projected/38e41b2b-5357-4e1b-8db3-837b19953be0-kube-api-access-n8hkq\") pod \"cert-manager-597b95cb84-l9fr6\" (UID: \"38e41b2b-5357-4e1b-8db3-837b19953be0\") " pod="cert-manager/cert-manager-597b95cb84-l9fr6" Oct 30 21:22:03.816968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.816944 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38e41b2b-5357-4e1b-8db3-837b19953be0-bound-sa-token\") pod \"cert-manager-597b95cb84-l9fr6\" (UID: \"38e41b2b-5357-4e1b-8db3-837b19953be0\") " pod="cert-manager/cert-manager-597b95cb84-l9fr6" Oct 30 21:22:03.845032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:03.844991 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-l9fr6" Oct 30 21:22:04.230367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:04.230341 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-l9fr6"] Oct 30 21:22:04.233351 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:22:04.233325 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38e41b2b_5357_4e1b_8db3_837b19953be0.slice/crio-a16479e444ad99df5216532a89d10c2fdd4268d1e63ac1dd0af853d727036343 WatchSource:0}: Error finding container a16479e444ad99df5216532a89d10c2fdd4268d1e63ac1dd0af853d727036343: Status 404 returned error can't find the container with id a16479e444ad99df5216532a89d10c2fdd4268d1e63ac1dd0af853d727036343 Oct 30 21:22:05.181142 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:05.181106 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-l9fr6" event={"ID":"38e41b2b-5357-4e1b-8db3-837b19953be0","Type":"ContainerStarted","Data":"4624e57e05c824b809adbd6ec79ace3d370693e167eb09dfc081f1cc3ea118c4"} Oct 30 21:22:05.181615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:05.181150 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-l9fr6" event={"ID":"38e41b2b-5357-4e1b-8db3-837b19953be0","Type":"ContainerStarted","Data":"a16479e444ad99df5216532a89d10c2fdd4268d1e63ac1dd0af853d727036343"} Oct 30 21:22:05.293682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:05.293616 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-l9fr6" podStartSLOduration=2.293595102 podStartE2EDuration="2.293595102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:05.293271901 +0000 UTC m=+315.426645101" watchObservedRunningTime="2026-10-30 21:22:05.293595102 +0000 UTC m=+315.426968301" Oct 30 21:22:08.175927 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:08.175896 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" Oct 30 21:22:12.476416 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:22:12.476378 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:22:16.961143 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:16.961110 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv"] Oct 30 21:22:16.967691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:16.967661 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.022614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.022583 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv"] Oct 30 21:22:17.067692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.067657 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f50aabf8-5555-49b7-ac58-a6370c512b49-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.067898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.067733 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wjh\" (UniqueName: \"kubernetes.io/projected/f50aabf8-5555-49b7-ac58-a6370c512b49-kube-api-access-85wjh\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.067973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.067916 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f50aabf8-5555-49b7-ac58-a6370c512b49-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.070739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.070720 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:22:17.070739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.070730 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 21:22:17.070861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.070743 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8m46b\"" Oct 30 21:22:17.070861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.070747 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 21:22:17.070861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.070724 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:22:17.168703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.168669 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wjh\" (UniqueName: \"kubernetes.io/projected/f50aabf8-5555-49b7-ac58-a6370c512b49-kube-api-access-85wjh\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.168889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.168726 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f50aabf8-5555-49b7-ac58-a6370c512b49-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.168889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.168755 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f50aabf8-5555-49b7-ac58-a6370c512b49-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.169383 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.169367 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f50aabf8-5555-49b7-ac58-a6370c512b49-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.171532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.171511 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f50aabf8-5555-49b7-ac58-a6370c512b49-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.225865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.225795 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wjh\" (UniqueName: \"kubernetes.io/projected/f50aabf8-5555-49b7-ac58-a6370c512b49-kube-api-access-85wjh\") pod \"kuadrant-console-plugin-66c4fb575d-nsqsv\" (UID: \"f50aabf8-5555-49b7-ac58-a6370c512b49\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.279602 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.279564 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" Oct 30 21:22:17.652240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:17.652209 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv"] Oct 30 21:22:17.655346 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:22:17.655320 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf50aabf8_5555_49b7_ac58_a6370c512b49.slice/crio-93773708f822908fa6695cb24aec017d3e1c31c7128bdb70f943520cddbd4511 WatchSource:0}: Error finding container 93773708f822908fa6695cb24aec017d3e1c31c7128bdb70f943520cddbd4511: Status 404 returned error can't find the container with id 93773708f822908fa6695cb24aec017d3e1c31c7128bdb70f943520cddbd4511 Oct 30 21:22:18.227979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:18.227937 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" event={"ID":"f50aabf8-5555-49b7-ac58-a6370c512b49","Type":"ContainerStarted","Data":"93773708f822908fa6695cb24aec017d3e1c31c7128bdb70f943520cddbd4511"} Oct 30 21:22:24.254016 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:24.253981 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" event={"ID":"f50aabf8-5555-49b7-ac58-a6370c512b49","Type":"ContainerStarted","Data":"a945f4ba79f93507d4de92314daa49251cd4c374a0c1c2fa3319fda33f05f652"} Oct 30 21:22:24.376098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:24.376034 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nsqsv" podStartSLOduration=2.778966294 podStartE2EDuration="8.376012496s" totalImagesPullingTime="5.597046202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:24.375413831 +0000 UTC m=+334.508787028" watchObservedRunningTime="2026-10-30 21:22:24.376012496 +0000 UTC m=+334.509385695" Oct 30 21:22:42.475382 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:22:42.475343 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:22:42.691252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.691223 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-578b8bf98-fgttk"] Oct 30 21:22:42.695554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.695531 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.747948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.747845 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:22:42.748175 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.747852 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:22:42.748175 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.747885 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:22:42.783429 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.783395 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5075246-d0cb-4e2b-af88-559679cf87bb-tmp\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.783615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.783437 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2nr\" (UniqueName: \"kubernetes.io/projected/d5075246-d0cb-4e2b-af88-559679cf87bb-kube-api-access-dz2nr\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.783615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.783561 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5075246-d0cb-4e2b-af88-559679cf87bb-tls-certs\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.797460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.797435 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2hw5t\"" Oct 30 21:22:42.797460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.797460 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:22:42.798740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.798720 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-578b8bf98-fgttk"] Oct 30 21:22:42.884465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.884429 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5075246-d0cb-4e2b-af88-559679cf87bb-tls-certs\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.884641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.884510 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5075246-d0cb-4e2b-af88-559679cf87bb-tmp\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.884641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.884554 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2nr\" (UniqueName: \"kubernetes.io/projected/d5075246-d0cb-4e2b-af88-559679cf87bb-kube-api-access-dz2nr\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.887058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.887028 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5075246-d0cb-4e2b-af88-559679cf87bb-tmp\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.887171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.887152 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5075246-d0cb-4e2b-af88-559679cf87bb-tls-certs\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:42.941353 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:42.941322 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2nr\" (UniqueName: \"kubernetes.io/projected/d5075246-d0cb-4e2b-af88-559679cf87bb-kube-api-access-dz2nr\") pod \"kube-auth-proxy-578b8bf98-fgttk\" (UID: \"d5075246-d0cb-4e2b-af88-559679cf87bb\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:43.008719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:43.008628 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" Oct 30 21:22:43.381343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:43.381310 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-578b8bf98-fgttk"] Oct 30 21:22:43.385045 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:22:43.385016 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5075246_d0cb_4e2b_af88_559679cf87bb.slice/crio-8368a44b2b140c739354e88254a3e0a7ba7cf8a5da12f72fc0f0b79b688e2a25 WatchSource:0}: Error finding container 8368a44b2b140c739354e88254a3e0a7ba7cf8a5da12f72fc0f0b79b688e2a25: Status 404 returned error can't find the container with id 8368a44b2b140c739354e88254a3e0a7ba7cf8a5da12f72fc0f0b79b688e2a25 Oct 30 21:22:44.334836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:44.334797 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" event={"ID":"d5075246-d0cb-4e2b-af88-559679cf87bb","Type":"ContainerStarted","Data":"8368a44b2b140c739354e88254a3e0a7ba7cf8a5da12f72fc0f0b79b688e2a25"} Oct 30 21:22:45.764298 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.764263 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7"] Oct 30 21:22:45.770861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.770824 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.803890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.803842 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvgj\" (UniqueName: \"kubernetes.io/projected/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-kube-api-access-7xvgj\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.804068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.803904 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-metrics-certs\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.804068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.804016 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-cert\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.804193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.804097 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-manager-config\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.824533 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.824482 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:22:45.873826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.873792 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 21:22:45.874038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.873898 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:22:45.874119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.874038 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-9x4w7\"" Oct 30 21:22:45.874119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.874088 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 21:22:45.874218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.874118 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 21:22:45.875595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.875562 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7"] Oct 30 21:22:45.904862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.904826 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvgj\" (UniqueName: \"kubernetes.io/projected/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-kube-api-access-7xvgj\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.905046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.904879 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-metrics-certs\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.905046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.904915 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-cert\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.905046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.904943 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-manager-config\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.905707 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.905683 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-manager-config\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.907609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.907590 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-cert\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.907721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.907648 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-metrics-certs\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:45.961989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:45.961963 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvgj\" (UniqueName: \"kubernetes.io/projected/12148a19-ecf9-44f3-a8b8-dbbaa0a568b5-kube-api-access-7xvgj\") pod \"jobset-controller-manager-7b69d77886-7zjc7\" (UID: \"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:46.084183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:46.084135 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:46.470192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:46.465305 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7"] Oct 30 21:22:47.348396 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:47.348361 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" event={"ID":"d5075246-d0cb-4e2b-af88-559679cf87bb","Type":"ContainerStarted","Data":"635a2ac4c643c21bb81bab0d7c99642c5d827e8fea5c6a07d1a396dca36ee7c7"} Oct 30 21:22:47.349629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:47.349603 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" event={"ID":"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5","Type":"ContainerStarted","Data":"2d0722488e0fc7f48cd5932e1a0d504eaa97702fd8fbafcf35021188cd4cfdf5"} Oct 30 21:22:47.467034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:47.466970 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-578b8bf98-fgttk" podStartSLOduration=2.536543461 podStartE2EDuration="5.466947499s" totalImagesPullingTime="2.930404038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:47.46543592 +0000 UTC m=+357.598809118" watchObservedRunningTime="2026-10-30 21:22:47.466947499 +0000 UTC m=+357.600320697" Oct 30 21:22:50.362889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:50.362854 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" event={"ID":"12148a19-ecf9-44f3-a8b8-dbbaa0a568b5","Type":"ContainerStarted","Data":"d946607a9992a324d049a9895751a28ce6c886ab463f4bd5dba14e693df6c3ac"} Oct 30 21:22:50.363275 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:50.362958 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:22:50.512620 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:22:50.512509 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" podStartSLOduration=1.747246165 podStartE2EDuration="5.512475418s" totalImagesPullingTime="3.765229253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:50.511596989 +0000 UTC m=+360.644970198" watchObservedRunningTime="2026-10-30 21:22:50.512475418 +0000 UTC m=+360.645848617" Oct 30 21:23:01.372867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:23:01.372824 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7b69d77886-7zjc7" Oct 30 21:23:12.476396 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:23:12.476360 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:23:42.475890 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:23:42.475806 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:24:02.139764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.139715 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cnpp7"] Oct 30 21:24:02.142991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.142974 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:02.195072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.195036 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:24:02.244263 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.244226 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:24:02.244263 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.244256 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:24:02.244563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.244398 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-g7qb8\"" Oct 30 21:24:02.245781 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.245759 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cnpp7"] Oct 30 21:24:02.268662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.268629 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd81cee0-0dc6-4f42-8a61-3b5780e223bf-tls-certs\") pod \"model-serving-api-75bd89fdfc-cnpp7\" (UID: \"bd81cee0-0dc6-4f42-8a61-3b5780e223bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:02.268856 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.268695 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95lt\" (UniqueName: \"kubernetes.io/projected/bd81cee0-0dc6-4f42-8a61-3b5780e223bf-kube-api-access-b95lt\") pod \"model-serving-api-75bd89fdfc-cnpp7\" (UID: \"bd81cee0-0dc6-4f42-8a61-3b5780e223bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:02.369387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.369351 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95lt\" (UniqueName: \"kubernetes.io/projected/bd81cee0-0dc6-4f42-8a61-3b5780e223bf-kube-api-access-b95lt\") pod \"model-serving-api-75bd89fdfc-cnpp7\" (UID: \"bd81cee0-0dc6-4f42-8a61-3b5780e223bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:02.369618 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.369409 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd81cee0-0dc6-4f42-8a61-3b5780e223bf-tls-certs\") pod \"model-serving-api-75bd89fdfc-cnpp7\" (UID: \"bd81cee0-0dc6-4f42-8a61-3b5780e223bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:02.372015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.371991 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd81cee0-0dc6-4f42-8a61-3b5780e223bf-tls-certs\") pod \"model-serving-api-75bd89fdfc-cnpp7\" (UID: \"bd81cee0-0dc6-4f42-8a61-3b5780e223bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:02.426672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.426600 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95lt\" (UniqueName: \"kubernetes.io/projected/bd81cee0-0dc6-4f42-8a61-3b5780e223bf-kube-api-access-b95lt\") pod \"model-serving-api-75bd89fdfc-cnpp7\" (UID: \"bd81cee0-0dc6-4f42-8a61-3b5780e223bf\") " pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:02.455654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.455621 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:02.824189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:02.824151 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cnpp7"] Oct 30 21:24:02.827665 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:24:02.827640 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd81cee0_0dc6_4f42_8a61_3b5780e223bf.slice/crio-9cfa29a5f5e2ee749730b704bbbed144a5d2652a291194d7506df22e87a48e7b WatchSource:0}: Error finding container 9cfa29a5f5e2ee749730b704bbbed144a5d2652a291194d7506df22e87a48e7b: Status 404 returned error can't find the container with id 9cfa29a5f5e2ee749730b704bbbed144a5d2652a291194d7506df22e87a48e7b Oct 30 21:24:03.653389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:03.653356 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" event={"ID":"bd81cee0-0dc6-4f42-8a61-3b5780e223bf","Type":"ContainerStarted","Data":"9cfa29a5f5e2ee749730b704bbbed144a5d2652a291194d7506df22e87a48e7b"} Oct 30 21:24:05.663843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:05.663798 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" event={"ID":"bd81cee0-0dc6-4f42-8a61-3b5780e223bf","Type":"ContainerStarted","Data":"9b7df472c9c13acde0366ac7c0796570b824c2220e73f4e84def56a49d64c1ac"} Oct 30 21:24:05.664309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:05.663939 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:05.779482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:05.779425 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" podStartSLOduration=1.7532693529999999 podStartE2EDuration="3.77940863s" totalImagesPullingTime="2.026139277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:05.778233843 +0000 UTC m=+435.911607042" watchObservedRunningTime="2026-10-30 21:24:05.77940863 +0000 UTC m=+435.912781827" Oct 30 21:24:12.479333 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:24:12.479288 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:24:16.672431 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:16.672401 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-cnpp7" Oct 30 21:24:28.214719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:24:28.214675 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podUID="a4207178a950303c256d4f333088d96f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:42.475879 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:24:42.475837 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:25:12.476967 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:25:12.476883 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:25:42.476735 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:25:42.476695 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:26:09.558989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.558953 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z"] Oct 30 21:26:09.562154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.562135 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.627759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.627729 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.627926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.627761 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.627926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.627805 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.627926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.627882 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.628023 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.627972 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.628023 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.628000 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8qg\" (UniqueName: \"kubernetes.io/projected/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-kube-api-access-kf8qg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.636948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.636912 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:26:09.663760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.663724 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:26:09.663957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.663946 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 21:26:09.664030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.663978 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tv6dw\"" Oct 30 21:26:09.665227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.665204 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z"] Oct 30 21:26:09.729062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729021 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.729062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729063 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8qg\" (UniqueName: \"kubernetes.io/projected/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-kube-api-access-kf8qg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.729318 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729095 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.729318 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729120 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.729318 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729167 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.729318 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729194 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.729572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729554 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.729639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729593 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.729691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.729676 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.731661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.731640 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.731838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.731822 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.787191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.787150 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8qg\" (UniqueName: \"kubernetes.io/projected/3f6bfaff-39af-4c61-a3b5-5d25d0098a86-kube-api-access-kf8qg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z\" (UID: \"3f6bfaff-39af-4c61-a3b5-5d25d0098a86\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:09.874116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:09.874024 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:10.248257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:10.248178 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z"] Oct 30 21:26:10.251446 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:26:10.251410 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6bfaff_39af_4c61_a3b5_5d25d0098a86.slice/crio-aeb5166947f8544c24f89a3b02ca5279b9def164cb1753181eead71b8c436f89 WatchSource:0}: Error finding container aeb5166947f8544c24f89a3b02ca5279b9def164cb1753181eead71b8c436f89: Status 404 returned error can't find the container with id aeb5166947f8544c24f89a3b02ca5279b9def164cb1753181eead71b8c436f89 Oct 30 21:26:11.165776 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:11.165741 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" event={"ID":"3f6bfaff-39af-4c61-a3b5-5d25d0098a86","Type":"ContainerStarted","Data":"aeb5166947f8544c24f89a3b02ca5279b9def164cb1753181eead71b8c436f89"} Oct 30 21:26:12.171346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:12.171301 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" event={"ID":"3f6bfaff-39af-4c61-a3b5-5d25d0098a86","Type":"ContainerStarted","Data":"1f08b38b8e4c9812122189691edffe0b9e7e2294ee56ad8ad5b13d0faee2d46e"} Oct 30 21:26:12.288886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:12.288830 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" podStartSLOduration=1.758749816 podStartE2EDuration="3.288814401s" totalImagesPullingTime="1.530064585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:12.286882558 +0000 UTC m=+562.420255756" watchObservedRunningTime="2026-10-30 21:26:12.288814401 +0000 UTC m=+562.422187598" Oct 30 21:26:12.478248 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:26:12.478115 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:26:12.874701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:12.874676 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:13.525389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.525344 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq"] Oct 30 21:26:13.528700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.528677 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.629605 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.629576 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:26:13.631166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.631147 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq"] Oct 30 21:26:13.664881 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.664847 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.665048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.664901 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.665048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.664971 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.665048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.665009 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.665048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.665029 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.665182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.665119 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qmx\" (UniqueName: \"kubernetes.io/projected/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-kube-api-access-d7qmx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766085 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qmx\" (UniqueName: \"kubernetes.io/projected/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-kube-api-access-d7qmx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766145 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766181 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766198 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766221 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766235 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766607 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766678 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.766713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.766698 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.768660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.768632 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.769059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.769039 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.823980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.823943 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qmx\" (UniqueName: \"kubernetes.io/projected/c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b-kube-api-access-d7qmx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq\" (UID: \"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:13.840859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:13.840824 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:14.212913 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:14.212835 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq"] Oct 30 21:26:14.216462 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:26:14.216435 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc58e5a64_1b9f_4ba4_8f36_c7a59ccc066b.slice/crio-c81b0856b58fb830b35047d2b2768e51887c6e3bb9befbe910280a4a8678a572 WatchSource:0}: Error finding container c81b0856b58fb830b35047d2b2768e51887c6e3bb9befbe910280a4a8678a572: Status 404 returned error can't find the container with id c81b0856b58fb830b35047d2b2768e51887c6e3bb9befbe910280a4a8678a572 Oct 30 21:26:15.183621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:15.183536 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" event={"ID":"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b","Type":"ContainerStarted","Data":"b2eb337df948712806726f7b1f76e340985d262ee4a40ac85e6fb540e456c528"} Oct 30 21:26:15.183621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:15.183582 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" event={"ID":"c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b","Type":"ContainerStarted","Data":"c81b0856b58fb830b35047d2b2768e51887c6e3bb9befbe910280a4a8678a572"} Oct 30 21:26:15.298655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:15.298585 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podStartSLOduration=1.702250751 podStartE2EDuration="2.298563134s" totalImagesPullingTime="596.312383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:15.297339725 +0000 UTC m=+565.430712924" watchObservedRunningTime="2026-10-30 21:26:15.298563134 +0000 UTC m=+565.431936324" Oct 30 21:26:15.841593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:15.841557 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:19.594880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.594846 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r"] Oct 30 21:26:19.598176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.598159 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.699215 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.699178 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 21:26:19.700731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.700705 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r"] Oct 30 21:26:19.717731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.717702 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.717861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.717754 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.717861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.717835 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f6ca98-2796-4b8a-acaf-ea95b126220e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.717936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.717863 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.717969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.717951 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmm6\" (UniqueName: \"kubernetes.io/projected/d4f6ca98-2796-4b8a-acaf-ea95b126220e-kube-api-access-2cmm6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.718001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.717992 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819064 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmm6\" (UniqueName: \"kubernetes.io/projected/d4f6ca98-2796-4b8a-acaf-ea95b126220e-kube-api-access-2cmm6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819117 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819164 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819214 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819247 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f6ca98-2796-4b8a-acaf-ea95b126220e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819278 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819696 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819800 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819725 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.819800 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.819756 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.821540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.821521 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4f6ca98-2796-4b8a-acaf-ea95b126220e-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.821847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.821827 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f6ca98-2796-4b8a-acaf-ea95b126220e-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.874295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.874208 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:19.877132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.877098 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmm6\" (UniqueName: \"kubernetes.io/projected/d4f6ca98-2796-4b8a-acaf-ea95b126220e-kube-api-access-2cmm6\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kk47r\" (UID: \"d4f6ca98-2796-4b8a-acaf-ea95b126220e\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:19.887771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.887749 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:19.910075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:19.910046 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:20.215606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:20.215517 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" Oct 30 21:26:20.282077 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:20.282048 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r"] Oct 30 21:26:20.285067 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:26:20.285034 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f6ca98_2796_4b8a_acaf_ea95b126220e.slice/crio-985fd90f552f384b4c54572a47996fddb8f759f507031ee60df1aae3a580065a WatchSource:0}: Error finding container 985fd90f552f384b4c54572a47996fddb8f759f507031ee60df1aae3a580065a: Status 404 returned error can't find the container with id 985fd90f552f384b4c54572a47996fddb8f759f507031ee60df1aae3a580065a Oct 30 21:26:21.209668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:21.209629 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" event={"ID":"d4f6ca98-2796-4b8a-acaf-ea95b126220e","Type":"ContainerStarted","Data":"1b3c979e048b87c475d0a8b5343fd3878f41e95ab35b090863162c5db5a35f69"} Oct 30 21:26:21.210113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:21.209693 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" event={"ID":"d4f6ca98-2796-4b8a-acaf-ea95b126220e","Type":"ContainerStarted","Data":"985fd90f552f384b4c54572a47996fddb8f759f507031ee60df1aae3a580065a"} Oct 30 21:26:21.326157 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:21.326104 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" podStartSLOduration=1.6005470609999999 podStartE2EDuration="2.326087615s" totalImagesPullingTime="725.540554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:21.324473069 +0000 UTC m=+571.457846266" watchObservedRunningTime="2026-10-30 21:26:21.326087615 +0000 UTC m=+571.459460812" Oct 30 21:26:21.910156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:21.910121 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:23.841542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:23.841507 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:23.854077 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:23.854053 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:24.233370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:24.233291 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:26:29.911277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:29.911235 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:29.924069 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:29.924041 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:30.258008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.257930 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kk47r" Oct 30 21:26:30.368111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.368076 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh"] Oct 30 21:26:30.371500 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.371471 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.371667 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.371650 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64"] Oct 30 21:26:30.374808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.374791 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.410368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410330 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.410368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410370 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb64d\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-kube-api-access-jb64d\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.410649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410440 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66449278-1a38-4a04-a93d-1fe119de3c81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.410649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410542 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.410649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410574 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.410819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410662 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.410819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410696 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.410819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410723 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66449278-1a38-4a04-a93d-1fe119de3c81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.410819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410798 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.410978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410824 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.410978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410855 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.410978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410926 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.410978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410953 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.410978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410968 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.411131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.410990 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj4qs\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-kube-api-access-gj4qs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.411131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.411075 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.411131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.411110 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.411131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.411126 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.423586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.423555 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:26:30.451475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.451443 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x8rmh\"" Oct 30 21:26:30.467757 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.467721 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh"] Oct 30 21:26:30.512192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512105 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512140 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512159 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512185 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj4qs\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-kube-api-access-gj4qs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512216 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512254 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512280 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512317 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512338 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb64d\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-kube-api-access-jb64d\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512517 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66449278-1a38-4a04-a93d-1fe119de3c81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512550 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512610 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512645 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512647 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512729 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512759 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66449278-1a38-4a04-a93d-1fe119de3c81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512759 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512799 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512807 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512830 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.512868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512864 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.513443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.512928 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.513443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.513020 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.513443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.513059 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.513443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.513238 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.513443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.513370 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.513719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.513467 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66449278-1a38-4a04-a93d-1fe119de3c81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.513719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.513635 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.514919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.514897 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.515189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.515175 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.516003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.515973 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66449278-1a38-4a04-a93d-1fe119de3c81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.516099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.516047 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.569655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.569626 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.569826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.569692 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb64d\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-kube-api-access-jb64d\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.569962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.569940 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bz6dh\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.570079 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.570061 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj4qs\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-kube-api-access-gj4qs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-flf64\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:30.576248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.576219 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64"] Oct 30 21:26:30.685579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.685547 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:30.691997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:30.691976 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:31.127982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:31.127955 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh"] Oct 30 21:26:31.130501 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:31.130384 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64"] Oct 30 21:26:31.131021 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:26:31.130954 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca52af0_9cb4_4353_98b1_92e69b6c5b9a.slice/crio-4871e50e7fb0c53efb540826b26994b70c14fac4610bf49edc966a393be6b20e WatchSource:0}: Error finding container 4871e50e7fb0c53efb540826b26994b70c14fac4610bf49edc966a393be6b20e: Status 404 returned error can't find the container with id 4871e50e7fb0c53efb540826b26994b70c14fac4610bf49edc966a393be6b20e Oct 30 21:26:31.133251 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:26:31.133223 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66449278_1a38_4a04_a93d_1fe119de3c81.slice/crio-e1bf3111411a892b0135b5a584548f5885fbdd80c4f55cd8af5b022cbc81fbe9 WatchSource:0}: Error finding container e1bf3111411a892b0135b5a584548f5885fbdd80c4f55cd8af5b022cbc81fbe9: Status 404 returned error can't find the container with id e1bf3111411a892b0135b5a584548f5885fbdd80c4f55cd8af5b022cbc81fbe9 Oct 30 21:26:31.251687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:31.251603 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" event={"ID":"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a","Type":"ContainerStarted","Data":"4871e50e7fb0c53efb540826b26994b70c14fac4610bf49edc966a393be6b20e"} Oct 30 21:26:31.252815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:31.252793 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" event={"ID":"66449278-1a38-4a04-a93d-1fe119de3c81","Type":"ContainerStarted","Data":"e1bf3111411a892b0135b5a584548f5885fbdd80c4f55cd8af5b022cbc81fbe9"} Oct 30 21:26:35.107538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.107476 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:35.107982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.107579 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:35.107982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.107611 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:35.273047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.273015 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" event={"ID":"66449278-1a38-4a04-a93d-1fe119de3c81","Type":"ContainerStarted","Data":"95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88"} Oct 30 21:26:35.284190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.284153 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:35.284286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.284237 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:35.284286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.284278 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:35.424854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.424762 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" podStartSLOduration=1.454022989 podStartE2EDuration="5.424746999s" totalImagesPullingTime="3.97072401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:35.388469947 +0000 UTC m=+585.521843146" watchObservedRunningTime="2026-10-30 21:26:35.424746999 +0000 UTC m=+585.558120193" Oct 30 21:26:35.692972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.692880 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:35.912512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.907005 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr"] Oct 30 21:26:35.912512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.911004 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:35.959578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.959299 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr"] Oct 30 21:26:35.963260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.963231 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 21:26:35.964694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.964664 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bjq\" (UniqueName: \"kubernetes.io/projected/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-kube-api-access-24bjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:35.964803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.964705 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:35.964803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.964727 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:35.964803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.964750 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:35.964803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.964770 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:35.964966 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:35.964804 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.065638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.065605 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.065815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.065646 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.065815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.065668 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.065815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.065707 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.065815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.065752 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bjq\" (UniqueName: \"kubernetes.io/projected/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-kube-api-access-24bjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.065815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.065772 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.066097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.066073 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.066155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.066084 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.066208 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.066187 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.068157 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.068140 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.068333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.068318 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.145976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.145933 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bjq\" (UniqueName: \"kubernetes.io/projected/c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32-kube-api-access-24bjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr\" (UID: \"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.234566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.231675 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:36.279389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.279339 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" event={"ID":"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a","Type":"ContainerStarted","Data":"794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f"} Oct 30 21:26:36.635255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.634144 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podStartSLOduration=2.48414241 podStartE2EDuration="6.634119439s" totalImagesPullingTime="4.149977029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:36.408518174 +0000 UTC m=+586.541891374" watchObservedRunningTime="2026-10-30 21:26:36.634119439 +0000 UTC m=+586.767492643" Oct 30 21:26:36.643562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.636677 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr"] Oct 30 21:26:36.644050 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:26:36.644017 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ee0ddf_4b0d_4a63_ad5b_a5dd6cc98a32.slice/crio-304b57cdeb141712ca5cc9a3be89d87c3013a946347c4a4d7319bd4b2464de57 WatchSource:0}: Error finding container 304b57cdeb141712ca5cc9a3be89d87c3013a946347c4a4d7319bd4b2464de57: Status 404 returned error can't find the container with id 304b57cdeb141712ca5cc9a3be89d87c3013a946347c4a4d7319bd4b2464de57 Oct 30 21:26:36.685810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.685765 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:36.687002 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.686977 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 21:26:36.687108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.687027 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 21:26:36.693085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.693061 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flf64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:26:36.693204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:36.693108 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:26:37.289750 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:37.289718 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" event={"ID":"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32","Type":"ContainerStarted","Data":"304b57cdeb141712ca5cc9a3be89d87c3013a946347c4a4d7319bd4b2464de57"} Oct 30 21:26:37.686280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:37.686187 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 21:26:37.686280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:37.686253 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 21:26:37.692653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:37.692630 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flf64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:26:37.692737 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:37.692679 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:26:38.301515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:38.293464 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" event={"ID":"c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32","Type":"ContainerStarted","Data":"cc8d83ced7a355d8d76197e1bc3425f872f2e5742ed3e0eab2005d66275ce668"} Oct 30 21:26:38.414245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:38.414182 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" podStartSLOduration=2.7758784629999997 podStartE2EDuration="3.414160128s" totalImagesPullingTime="638.281665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:38.412924538 +0000 UTC m=+588.546297737" watchObservedRunningTime="2026-10-30 21:26:38.414160128 +0000 UTC m=+588.547533329" Oct 30 21:26:38.686688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:38.686602 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 21:26:38.686688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:38.686662 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 21:26:38.692306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:38.692284 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flf64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:26:38.692351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:38.692328 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:26:39.233988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:39.233956 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:39.686576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:39.686326 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 21:26:39.686576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:39.686399 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 21:26:39.693035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:39.693004 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flf64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:26:39.693184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:39.693058 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:26:40.686147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:40.686118 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 21:26:40.686382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:40.686361 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 21:26:40.692620 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:40.692599 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flf64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:26:40.692930 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:40.692641 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:26:41.686636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:41.686600 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 21:26:41.686819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:41.686668 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 21:26:41.692891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:41.692869 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-flf64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Oct 30 21:26:41.693172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:41.692913 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Oct 30 21:26:42.476654 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:26:42.476613 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:26:42.686295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:42.686258 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Oct 30 21:26:42.686482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:42.686327 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Oct 30 21:26:43.295842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:43.295807 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:43.296762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:43.296585 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:43.297459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:43.297434 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:26:44.728550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:44.727687 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:44.728550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:44.727775 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:45.685845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:45.685802 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:45.769699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:45.769659 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bz6dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:45.770108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:45.769721 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:45.795070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:45.795036 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:46.232090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:46.232059 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:46.244650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:46.244621 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:26:46.354199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:46.354176 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:26:46.364619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:26:46.364592 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" Oct 30 21:27:06.283083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.283041 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72"] Oct 30 21:27:06.286692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.286667 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:06.338007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.337971 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:27:06.381074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.381040 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b88743-d84b-4095-be43-e1f3bbb50261-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-r8g72\" (UID: \"b7b88743-d84b-4095-be43-e1f3bbb50261\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:06.381236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.381087 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnm5h\" (UniqueName: \"kubernetes.io/projected/b7b88743-d84b-4095-be43-e1f3bbb50261-kube-api-access-gnm5h\") pod \"ai-gateway-controller-5f85cd9cfc-r8g72\" (UID: \"b7b88743-d84b-4095-be43-e1f3bbb50261\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:06.387432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.387414 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:27:06.387432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.387421 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xrgvt\"" Oct 30 21:27:06.389150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.389133 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72"] Oct 30 21:27:06.482379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.482346 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b88743-d84b-4095-be43-e1f3bbb50261-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-r8g72\" (UID: \"b7b88743-d84b-4095-be43-e1f3bbb50261\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:06.482592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.482404 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnm5h\" (UniqueName: \"kubernetes.io/projected/b7b88743-d84b-4095-be43-e1f3bbb50261-kube-api-access-gnm5h\") pod \"ai-gateway-controller-5f85cd9cfc-r8g72\" (UID: \"b7b88743-d84b-4095-be43-e1f3bbb50261\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:06.485012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.484982 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b88743-d84b-4095-be43-e1f3bbb50261-metrics-certs\") pod \"ai-gateway-controller-5f85cd9cfc-r8g72\" (UID: \"b7b88743-d84b-4095-be43-e1f3bbb50261\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:06.540976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.540946 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnm5h\" (UniqueName: \"kubernetes.io/projected/b7b88743-d84b-4095-be43-e1f3bbb50261-kube-api-access-gnm5h\") pod \"ai-gateway-controller-5f85cd9cfc-r8g72\" (UID: \"b7b88743-d84b-4095-be43-e1f3bbb50261\") " pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:06.599073 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.599030 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:06.969061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.968980 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72"] Oct 30 21:27:06.971940 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:27:06.971913 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b88743_d84b_4095_be43_e1f3bbb50261.slice/crio-e4174f27ab4558c612f291f3c7f37d79a5983170c9b7b7116ea98bcc7280813a WatchSource:0}: Error finding container e4174f27ab4558c612f291f3c7f37d79a5983170c9b7b7116ea98bcc7280813a: Status 404 returned error can't find the container with id e4174f27ab4558c612f291f3c7f37d79a5983170c9b7b7116ea98bcc7280813a Oct 30 21:27:06.973282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:06.973263 4526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:27:07.441452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:07.441418 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" event={"ID":"b7b88743-d84b-4095-be43-e1f3bbb50261","Type":"ContainerStarted","Data":"e4174f27ab4558c612f291f3c7f37d79a5983170c9b7b7116ea98bcc7280813a"} Oct 30 21:27:12.477399 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:27:12.477359 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:27:14.477522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:14.477468 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" event={"ID":"b7b88743-d84b-4095-be43-e1f3bbb50261","Type":"ContainerStarted","Data":"8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378"} Oct 30 21:27:14.478007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:14.477553 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:14.590718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:14.590669 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" podStartSLOduration=2.037033931 podStartE2EDuration="8.590653978s" totalImagesPullingTime="6.553620047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:14.588650112 +0000 UTC m=+624.722023309" watchObservedRunningTime="2026-10-30 21:27:14.590653978 +0000 UTC m=+624.724027175" Oct 30 21:27:25.484903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:25.484874 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:25.839596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:25.839563 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-695b99ffdc-mbbg7"] Oct 30 21:27:25.843074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:25.843057 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:25.899664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:25.899630 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-695b99ffdc-mbbg7"] Oct 30 21:27:25.965375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:25.965343 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c216f-e8e9-4a71-93fc-c4081601a2b6-metrics-certs\") pod \"ai-gateway-controller-695b99ffdc-mbbg7\" (UID: \"d44c216f-e8e9-4a71-93fc-c4081601a2b6\") " pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:25.965375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:25.965379 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gfp\" (UniqueName: \"kubernetes.io/projected/d44c216f-e8e9-4a71-93fc-c4081601a2b6-kube-api-access-m9gfp\") pod \"ai-gateway-controller-695b99ffdc-mbbg7\" (UID: \"d44c216f-e8e9-4a71-93fc-c4081601a2b6\") " pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:26.066198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:26.066167 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c216f-e8e9-4a71-93fc-c4081601a2b6-metrics-certs\") pod \"ai-gateway-controller-695b99ffdc-mbbg7\" (UID: \"d44c216f-e8e9-4a71-93fc-c4081601a2b6\") " pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:26.066198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:26.066200 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gfp\" (UniqueName: \"kubernetes.io/projected/d44c216f-e8e9-4a71-93fc-c4081601a2b6-kube-api-access-m9gfp\") pod \"ai-gateway-controller-695b99ffdc-mbbg7\" (UID: \"d44c216f-e8e9-4a71-93fc-c4081601a2b6\") " pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:26.068903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:26.068881 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c216f-e8e9-4a71-93fc-c4081601a2b6-metrics-certs\") pod \"ai-gateway-controller-695b99ffdc-mbbg7\" (UID: \"d44c216f-e8e9-4a71-93fc-c4081601a2b6\") " pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:26.122427 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:26.122355 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gfp\" (UniqueName: \"kubernetes.io/projected/d44c216f-e8e9-4a71-93fc-c4081601a2b6-kube-api-access-m9gfp\") pod \"ai-gateway-controller-695b99ffdc-mbbg7\" (UID: \"d44c216f-e8e9-4a71-93fc-c4081601a2b6\") " pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:26.156259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:26.156226 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:26.528642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:26.528442 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-695b99ffdc-mbbg7"] Oct 30 21:27:26.531236 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:27:26.531203 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44c216f_e8e9_4a71_93fc_c4081601a2b6.slice/crio-7c18bcbdce6f5330c191e4be4a8525d51d55cfdf4b0d5c794af77a2b7b11de06 WatchSource:0}: Error finding container 7c18bcbdce6f5330c191e4be4a8525d51d55cfdf4b0d5c794af77a2b7b11de06: Status 404 returned error can't find the container with id 7c18bcbdce6f5330c191e4be4a8525d51d55cfdf4b0d5c794af77a2b7b11de06 Oct 30 21:27:27.533863 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.533821 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" event={"ID":"d44c216f-e8e9-4a71-93fc-c4081601a2b6","Type":"ContainerStarted","Data":"708a4b20e5fa4cbcbb18356fbd6a6e7e57c32fe35d3c0ba3eb0684265662ac5c"} Oct 30 21:27:27.534269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.533929 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:27.534269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.533948 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" event={"ID":"d44c216f-e8e9-4a71-93fc-c4081601a2b6","Type":"ContainerStarted","Data":"7c18bcbdce6f5330c191e4be4a8525d51d55cfdf4b0d5c794af77a2b7b11de06"} Oct 30 21:27:27.648195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.648081 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" podStartSLOduration=1.873015704 podStartE2EDuration="2.648066413s" totalImagesPullingTime="775.050709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:27.646568577 +0000 UTC m=+637.779941776" watchObservedRunningTime="2026-10-30 21:27:27.648066413 +0000 UTC m=+637.781439611" Oct 30 21:27:27.764842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.764807 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9"] Oct 30 21:27:27.768227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.768211 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:27.820245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.820202 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8rdsw\"" Oct 30 21:27:27.820245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.820202 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:27:27.826336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.826254 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9"] Oct 30 21:27:27.883946 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.883899 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx62n\" (UniqueName: \"kubernetes.io/projected/1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de-kube-api-access-tx62n\") pod \"payload-pre-processing-7b98bb678c-r9sf9\" (UID: \"1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:27.884150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.884023 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r9sf9\" (UID: \"1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:27.985258 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.985168 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r9sf9\" (UID: \"1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:27.985622 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.985599 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx62n\" (UniqueName: \"kubernetes.io/projected/1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de-kube-api-access-tx62n\") pod \"payload-pre-processing-7b98bb678c-r9sf9\" (UID: \"1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:27.986082 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:27.986039 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r9sf9\" (UID: \"1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:28.041746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:28.041720 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx62n\" (UniqueName: \"kubernetes.io/projected/1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de-kube-api-access-tx62n\") pod \"payload-pre-processing-7b98bb678c-r9sf9\" (UID: \"1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:28.081232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:28.081192 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:28.453763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:28.453736 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9"] Oct 30 21:27:28.456135 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:27:28.456096 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af9bcd6_6df5_4f62_88d6_c1fe98dbb2de.slice/crio-ffe8368252bc7801ca73a3a26a286080b03b60414e81fab04a66131918fe87b6 WatchSource:0}: Error finding container ffe8368252bc7801ca73a3a26a286080b03b60414e81fab04a66131918fe87b6: Status 404 returned error can't find the container with id ffe8368252bc7801ca73a3a26a286080b03b60414e81fab04a66131918fe87b6 Oct 30 21:27:28.539591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:28.539550 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" event={"ID":"1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de","Type":"ContainerStarted","Data":"ffe8368252bc7801ca73a3a26a286080b03b60414e81fab04a66131918fe87b6"} Oct 30 21:27:30.814915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.814880 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x"] Oct 30 21:27:30.826924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.826892 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.864508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.864455 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4"] Oct 30 21:27:30.868182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.868159 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:30.915913 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.915866 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d055332f-4f2b-4d52-842b-163859de6009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.916112 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.915987 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.916112 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.916059 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.916241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.916109 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d055332f-4f2b-4d52-842b-163859de6009-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.916241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.916222 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d055332f-4f2b-4d52-842b-163859de6009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.916348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.916258 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.916348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.916281 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.916348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.916340 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtvc\" (UniqueName: \"kubernetes.io/projected/d055332f-4f2b-4d52-842b-163859de6009-kube-api-access-bmtvc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.916532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.916378 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:30.922593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.922550 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x"] Oct 30 21:27:30.979057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:30.979023 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4"] Oct 30 21:27:31.017718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.017683 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.017903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.017743 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d055332f-4f2b-4d52-842b-163859de6009-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.017903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.017802 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.017903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.017821 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.017903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.017849 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.018090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.017963 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d055332f-4f2b-4d52-842b-163859de6009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.017996 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018027 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d055332f-4f2b-4d52-842b-163859de6009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018055 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnhn7\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-kube-api-access-hnhn7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.018265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018097 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018156 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018181 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.018265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018199 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.018265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018234 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.018535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018281 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.018535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018323 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtvc\" (UniqueName: \"kubernetes.io/projected/d055332f-4f2b-4d52-842b-163859de6009-kube-api-access-bmtvc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018339 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018356 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018403 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018446 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018998 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018635 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018998 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018793 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d055332f-4f2b-4d52-842b-163859de6009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.018998 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.018914 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.020615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.020594 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d055332f-4f2b-4d52-842b-163859de6009-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.020890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.020873 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d055332f-4f2b-4d52-842b-163859de6009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.077050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.077019 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d055332f-4f2b-4d52-842b-163859de6009-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.077183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.077070 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtvc\" (UniqueName: \"kubernetes.io/projected/d055332f-4f2b-4d52-842b-163859de6009-kube-api-access-bmtvc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nfl6x\" (UID: \"d055332f-4f2b-4d52-842b-163859de6009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.119591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119564 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.119778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119620 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.119778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119641 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.119778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119664 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.119778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119706 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnhn7\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-kube-api-access-hnhn7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.119778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119739 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.119778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119755 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.119778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119774 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.120147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.119790 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.120279 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.120258 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.120559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.120542 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.120559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.120546 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.120843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.120822 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.121084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.121065 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.122571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.122552 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.123143 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.123124 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.142685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.142645 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:31.177416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.177381 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.177604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.177430 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnhn7\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-kube-api-access-hnhn7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tzxd4\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.183427 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.183251 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:31.557406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.557368 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" event={"ID":"1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de","Type":"ContainerStarted","Data":"9f31bf47e18614099e8f514b5d62cf7776d757d486e7f07eac95063d25739e31"} Oct 30 21:27:31.557596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.557465 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:27:31.569224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.569197 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x"] Oct 30 21:27:31.610391 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:27:31.610362 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd055332f_4f2b_4d52_842b_163859de6009.slice/crio-038a89c5cc851215d58a856bf33fdda6ca347272d1557447303ef7ec034cde04 WatchSource:0}: Error finding container 038a89c5cc851215d58a856bf33fdda6ca347272d1557447303ef7ec034cde04: Status 404 returned error can't find the container with id 038a89c5cc851215d58a856bf33fdda6ca347272d1557447303ef7ec034cde04 Oct 30 21:27:31.612630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.612600 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.612714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.612674 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.612714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.612705 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.622167 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.622133 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4"] Oct 30 21:27:31.624127 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:27:31.624103 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cdf6fb_6c69_4ad3_b49c_8388305fed43.slice/crio-bbac10b3513f64f86c95e774513aaae986a928dd852c374165987029134d79d8 WatchSource:0}: Error finding container bbac10b3513f64f86c95e774513aaae986a928dd852c374165987029134d79d8: Status 404 returned error can't find the container with id bbac10b3513f64f86c95e774513aaae986a928dd852c374165987029134d79d8 Oct 30 21:27:31.626377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.626347 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.626453 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.626421 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.626532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.626467 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.670628 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:31.669560 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" podStartSLOduration=2.06075541 podStartE2EDuration="4.669541256s" totalImagesPullingTime="2.608785846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:31.668894609 +0000 UTC m=+641.802268014" watchObservedRunningTime="2026-10-30 21:27:31.669541256 +0000 UTC m=+641.802914455" Oct 30 21:27:32.563568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:32.563532 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" event={"ID":"66cdf6fb-6c69-4ad3-b49c-8388305fed43","Type":"ContainerStarted","Data":"ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7"} Oct 30 21:27:32.563957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:32.563594 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" event={"ID":"66cdf6fb-6c69-4ad3-b49c-8388305fed43","Type":"ContainerStarted","Data":"bbac10b3513f64f86c95e774513aaae986a928dd852c374165987029134d79d8"} Oct 30 21:27:32.565114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:32.565094 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" event={"ID":"d055332f-4f2b-4d52-842b-163859de6009","Type":"ContainerStarted","Data":"f2612a46131fd94f91e06ffe7b6de2380fd8866cf29f0fecd477717f10f2c79c"} Oct 30 21:27:32.565218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:32.565131 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" event={"ID":"d055332f-4f2b-4d52-842b-163859de6009","Type":"ContainerStarted","Data":"038a89c5cc851215d58a856bf33fdda6ca347272d1557447303ef7ec034cde04"} Oct 30 21:27:32.731512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:32.730616 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podStartSLOduration=2.730594016 podStartE2EDuration="2.730594016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:32.728592443 +0000 UTC m=+642.861965644" watchObservedRunningTime="2026-10-30 21:27:32.730594016 +0000 UTC m=+642.863967215" Oct 30 21:27:32.845914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:32.845766 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podStartSLOduration=2.845748383 podStartE2EDuration="2.845748383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:32.845182133 +0000 UTC m=+642.978555332" watchObservedRunningTime="2026-10-30 21:27:32.845748383 +0000 UTC m=+642.979121582" Oct 30 21:27:33.143678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:33.143584 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:33.144566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:33.144537 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:27:33.144687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:33.144627 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 21:27:33.184077 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:33.184054 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:33.185386 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:33.185364 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tzxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:27:33.185507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:33.185422 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 21:27:34.144236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:34.144192 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:27:34.144813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:34.144276 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 21:27:34.183871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:34.183840 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tzxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:27:34.184048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:34.183904 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 21:27:35.143944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:35.143902 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:27:35.144168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:35.143957 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 21:27:35.184363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:35.184318 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tzxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:27:35.184819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:35.184375 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 21:27:36.143918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:36.143871 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:27:36.144108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:36.143930 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 21:27:36.184857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:36.184827 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tzxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:27:36.185369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:36.185341 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 21:27:37.144514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:37.143994 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:27:37.144514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:37.144059 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 21:27:37.184132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:37.184087 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tzxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:27:37.184289 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:37.184152 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 21:27:38.143636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:38.143593 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:27:38.144078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:38.143663 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 21:27:38.184401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:38.184360 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tzxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:27:38.184593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:38.184447 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 21:27:38.542705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:38.542672 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-695b99ffdc-mbbg7" Oct 30 21:27:38.708325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:38.708269 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72"] Oct 30 21:27:38.708939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:38.708906 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" podUID="b7b88743-d84b-4095-be43-e1f3bbb50261" containerName="manager" containerID="cri-o://8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378" gracePeriod=30 Oct 30 21:27:38.977064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:38.977040 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:39.143734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.143646 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:27:39.143734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.143700 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 21:27:39.152018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.151994 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7b88743-d84b-4095-be43-e1f3bbb50261-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b88743-d84b-4095-be43-e1f3bbb50261-metrics-certs\") pod \"b7b88743-d84b-4095-be43-e1f3bbb50261\" (UID: \"b7b88743-d84b-4095-be43-e1f3bbb50261\") " Oct 30 21:27:39.152094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.152084 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b88743-d84b-4095-be43-e1f3bbb50261-kube-api-access-gnm5h\" (UniqueName: \"kubernetes.io/projected/b7b88743-d84b-4095-be43-e1f3bbb50261-kube-api-access-gnm5h\") pod \"b7b88743-d84b-4095-be43-e1f3bbb50261\" (UID: \"b7b88743-d84b-4095-be43-e1f3bbb50261\") " Oct 30 21:27:39.154466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.154431 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b88743-d84b-4095-be43-e1f3bbb50261-kube-api-access-gnm5h" pod "b7b88743-d84b-4095-be43-e1f3bbb50261" (UID: "b7b88743-d84b-4095-be43-e1f3bbb50261"). InnerVolumeSpecName "kube-api-access-gnm5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:39.154753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.154729 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b88743-d84b-4095-be43-e1f3bbb50261-metrics-certs" pod "b7b88743-d84b-4095-be43-e1f3bbb50261" (UID: "b7b88743-d84b-4095-be43-e1f3bbb50261"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:39.184234 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.184207 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tzxd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:27:39.184340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.184251 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 21:27:39.253665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.253631 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnm5h\" (UniqueName: \"kubernetes.io/projected/b7b88743-d84b-4095-be43-e1f3bbb50261-kube-api-access-gnm5h\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:39.253665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.253660 4526 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b88743-d84b-4095-be43-e1f3bbb50261-metrics-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:27:39.602825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.602785 4526 generic.go:356] "Generic (PLEG): container finished" podID="b7b88743-d84b-4095-be43-e1f3bbb50261" containerID="8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378" exitCode=0 Oct 30 21:27:39.602825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.602835 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" event={"ID":"b7b88743-d84b-4095-be43-e1f3bbb50261","Type":"ContainerDied","Data":"8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378"} Oct 30 21:27:39.603109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.602852 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" Oct 30 21:27:39.603109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.602885 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72" event={"ID":"b7b88743-d84b-4095-be43-e1f3bbb50261","Type":"ContainerDied","Data":"e4174f27ab4558c612f291f3c7f37d79a5983170c9b7b7116ea98bcc7280813a"} Oct 30 21:27:39.603109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.602916 4526 scope.go:122] "RemoveContainer" containerID="8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378" Oct 30 21:27:39.616114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.616053 4526 scope.go:122] "RemoveContainer" containerID="8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378" Oct 30 21:27:39.616518 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:27:39.616467 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378\": container with ID starting with 8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378 not found: ID does not exist" containerID="8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378" Oct 30 21:27:39.616609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.616524 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378"} err="failed to get container status \"8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378\": rpc error: code = NotFound desc = could not find container \"8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378\": container with ID starting with 8d960a1ffcf99508c0cdb59c45b1857940ea568702d7c1a2865e3028c9f98378 not found: ID does not exist" Oct 30 21:27:39.766463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.766427 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72"] Oct 30 21:27:39.815365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.815328 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f85cd9cfc-r8g72"] Oct 30 21:27:39.987298 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:39.987202 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b88743-d84b-4095-be43-e1f3bbb50261" path="/var/lib/kubelet/pods/b7b88743-d84b-4095-be43-e1f3bbb50261/volumes" Oct 30 21:27:40.147757 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:40.147729 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:40.148117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:40.147987 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:40.148772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:40.148755 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" Oct 30 21:27:40.187432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:40.187407 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:40.187720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:40.187704 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:40.188455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:40.188442 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:27:42.477842 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:27:42.477803 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:27:42.569048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:27:42.569020 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r9sf9" Oct 30 21:28:12.476710 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:28:12.476626 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:28:28.215205 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:28.215163 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podUID="a4207178a950303c256d4f333088d96f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 21:28:42.477338 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:28:42.477306 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:28:47.826936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.826898 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm"] Oct 30 21:28:47.827568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.827458 4526 cpu_manager.go:414] "containerMap: removing container" podUID="b7b88743-d84b-4095-be43-e1f3bbb50261" containerName="manager" Oct 30 21:28:47.827568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.827479 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b88743-d84b-4095-be43-e1f3bbb50261" containerName="manager" Oct 30 21:28:47.827702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.827621 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b88743-d84b-4095-be43-e1f3bbb50261" containerName="manager" Oct 30 21:28:47.830692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.830671 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.890744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.890716 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-021907-openshift-default-dockercfg-dm7nx\"" Oct 30 21:28:47.934914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.934873 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27pdv\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-kube-api-access-27pdv\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.935113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.934969 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.935113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.935002 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.935113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.935030 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46444220-52f6-4122-b602-6a2a6531a4de-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.935113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.935064 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46444220-52f6-4122-b602-6a2a6531a4de-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.935113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.935104 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.935346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.935148 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.935346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.935313 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.935431 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.935346 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:47.942138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:47.942110 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm"] Oct 30 21:28:48.036902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.036866 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.036920 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.036953 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.036982 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037039 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27pdv\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-kube-api-access-27pdv\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037099 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037123 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037142 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46444220-52f6-4122-b602-6a2a6531a4de-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037177 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46444220-52f6-4122-b602-6a2a6531a4de-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037321 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037505 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037604 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.037878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.037852 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.038432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.038401 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46444220-52f6-4122-b602-6a2a6531a4de-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.039768 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.039750 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46444220-52f6-4122-b602-6a2a6531a4de-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.039887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.039844 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.094330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.094255 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.094468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.094406 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27pdv\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-kube-api-access-27pdv\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.143136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.143103 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:48.606728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.606704 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm"] Oct 30 21:28:48.609189 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:28:48.609165 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46444220_52f6_4122_b602_6a2a6531a4de.slice/crio-72f80181d0a56abed91c560f9a40e1e79ef8243c6ccc210e170cc266dca19206 WatchSource:0}: Error finding container 72f80181d0a56abed91c560f9a40e1e79ef8243c6ccc210e170cc266dca19206: Status 404 returned error can't find the container with id 72f80181d0a56abed91c560f9a40e1e79ef8243c6ccc210e170cc266dca19206 Oct 30 21:28:48.611356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.611325 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:48.611417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.611389 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:48.611452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.611417 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:48.751048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.751018 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846"] Oct 30 21:28:48.754617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.754595 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.806240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.806213 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-446ca2-openshift-default-dockercfg-2jv4g\"" Oct 30 21:28:48.815056 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.815023 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846"] Oct 30 21:28:48.844547 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.844503 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.845065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.844625 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.845065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.844662 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.845065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.844742 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hc5\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-kube-api-access-l6hc5\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.845065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.844792 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.845065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.844852 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.845065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.844885 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.845065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.844968 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.845065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.845038 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.916858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.916761 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" event={"ID":"46444220-52f6-4122-b602-6a2a6531a4de","Type":"ContainerStarted","Data":"4aecfe60cd248537425ce239a0062ca1a0c1ae43814424e8b120108aca8274c4"} Oct 30 21:28:48.916858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.916835 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" event={"ID":"46444220-52f6-4122-b602-6a2a6531a4de","Type":"ContainerStarted","Data":"72f80181d0a56abed91c560f9a40e1e79ef8243c6ccc210e170cc266dca19206"} Oct 30 21:28:48.946078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946045 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946087 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946106 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hc5\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-kube-api-access-l6hc5\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946125 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946152 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946180 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946223 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946286 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946359 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.946630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.946597 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.947333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.947298 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.947456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.947359 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.947555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.947538 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.948239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.948211 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.949042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.949020 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:48.950428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:48.950402 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:49.002501 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.002444 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:49.002676 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.002629 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hc5\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-kube-api-access-l6hc5\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:49.039695 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.039644 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" podStartSLOduration=2.039627889 podStartE2EDuration="2.039627889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:49.037854145 +0000 UTC m=+719.171227347" watchObservedRunningTime="2026-10-30 21:28:49.039627889 +0000 UTC m=+719.173001153" Oct 30 21:28:49.065445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.065408 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc"] Oct 30 21:28:49.068514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.068465 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:49.070116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.070098 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.123874 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.123843 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-07fa8e-openshift-default-dockercfg-whqhj\"" Oct 30 21:28:49.135882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.135848 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc"] Oct 30 21:28:49.144389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.144360 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:49.148136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148110 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda6d95a-c501-4387-b6c8-694af23150e6-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.148272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148150 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w75g2\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-kube-api-access-w75g2\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.148272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148237 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.148417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148279 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda6d95a-c501-4387-b6c8-694af23150e6-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.148417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148305 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.148417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148384 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.148603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148467 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.148675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148606 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.148675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.148637 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.150364 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.150343 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:49.250124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250042 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250084 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda6d95a-c501-4387-b6c8-694af23150e6-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250219 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w75g2\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-kube-api-access-w75g2\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250271 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250316 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda6d95a-c501-4387-b6c8-694af23150e6-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250341 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250369 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250458 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250555 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250705 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.250908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.250885 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda6d95a-c501-4387-b6c8-694af23150e6-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.251180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.251138 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.251180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.251150 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.251180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.251176 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.253065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.253035 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.253561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.253544 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda6d95a-c501-4387-b6c8-694af23150e6-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.311938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.311906 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.312087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.312021 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w75g2\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-kube-api-access-w75g2\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.403024 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.402984 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:49.454146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.453235 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846"] Oct 30 21:28:49.458894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.458408 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.458894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.458481 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.458894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.458535 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.777899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.777834 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc"] Oct 30 21:28:49.780359 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:28:49.780307 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda6d95a_c501_4387_b6c8_694af23150e6.slice/crio-db17289e0c4df6ff67f78cf2b1fb9e884b3ff68326941275e74b6428e13bceb7 WatchSource:0}: Error finding container db17289e0c4df6ff67f78cf2b1fb9e884b3ff68326941275e74b6428e13bceb7: Status 404 returned error can't find the container with id db17289e0c4df6ff67f78cf2b1fb9e884b3ff68326941275e74b6428e13bceb7 Oct 30 21:28:49.783007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.782964 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.783093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.783062 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.783130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.783107 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.924271 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.924237 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" event={"ID":"cda6d95a-c501-4387-b6c8-694af23150e6","Type":"ContainerStarted","Data":"7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c"} Oct 30 21:28:49.924813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.924293 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" event={"ID":"cda6d95a-c501-4387-b6c8-694af23150e6","Type":"ContainerStarted","Data":"db17289e0c4df6ff67f78cf2b1fb9e884b3ff68326941275e74b6428e13bceb7"} Oct 30 21:28:49.925894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.925867 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" event={"ID":"61876d79-62d0-49d1-9f3a-8e78d0e1e67b","Type":"ContainerStarted","Data":"68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee"} Oct 30 21:28:49.926018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.925911 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" event={"ID":"61876d79-62d0-49d1-9f3a-8e78d0e1e67b","Type":"ContainerStarted","Data":"e439145c0a0c5e7a474e1da4c96d85f8aea4d5bc62b430bdbd4dd98001152cf7"} Oct 30 21:28:49.926137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.926120 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:49.926941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:49.926918 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:28:50.042361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.042308 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" podStartSLOduration=1.042293708 podStartE2EDuration="1.042293708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:50.039420455 +0000 UTC m=+720.172793654" watchObservedRunningTime="2026-10-30 21:28:50.042293708 +0000 UTC m=+720.175666905" Oct 30 21:28:50.069553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.069527 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:50.074477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.074452 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:50.331603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.331481 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" podStartSLOduration=2.331467485 podStartE2EDuration="2.331467485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:50.329351218 +0000 UTC m=+720.462724420" watchObservedRunningTime="2026-10-30 21:28:50.331467485 +0000 UTC m=+720.464840682" Oct 30 21:28:50.403424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.403385 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:50.408743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.408714 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:50.931156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.931122 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:28:50.931678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.931175 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:50.932088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.932070 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:28:50.932173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:28:50.932119 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:29:02.090633 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.090598 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv"] Oct 30 21:29:02.094532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.094477 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:02.192422 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.192351 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv\" (UID: \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:02.193608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.193564 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pq92\" (UniqueName: \"kubernetes.io/projected/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-kube-api-access-6pq92\") pod \"payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv\" (UID: \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:02.197025 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.197000 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9796f1-dockercfg-nnt7f\"" Oct 30 21:29:02.197366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.197348 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9796f1\"" Oct 30 21:29:02.201503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.201455 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv"] Oct 30 21:29:02.248358 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.248329 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g"] Oct 30 21:29:02.252307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.252284 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:02.295249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.295207 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv\" (UID: \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:02.295249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.295253 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26jpc\" (UniqueName: \"kubernetes.io/projected/a79f6755-8dab-4c54-b8e1-db5a5e032726-kube-api-access-26jpc\") pod \"payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g\" (UID: \"a79f6755-8dab-4c54-b8e1-db5a5e032726\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:02.295474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.295316 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79f6755-8dab-4c54-b8e1-db5a5e032726-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g\" (UID: \"a79f6755-8dab-4c54-b8e1-db5a5e032726\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:02.295474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.295462 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pq92\" (UniqueName: \"kubernetes.io/projected/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-kube-api-access-6pq92\") pod \"payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv\" (UID: \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:02.295985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.295969 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv\" (UID: \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:02.302371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.302343 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g"] Oct 30 21:29:02.353115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.353040 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pq92\" (UniqueName: \"kubernetes.io/projected/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-kube-api-access-6pq92\") pod \"payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv\" (UID: \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:02.396748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.396697 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26jpc\" (UniqueName: \"kubernetes.io/projected/a79f6755-8dab-4c54-b8e1-db5a5e032726-kube-api-access-26jpc\") pod \"payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g\" (UID: \"a79f6755-8dab-4c54-b8e1-db5a5e032726\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:02.396748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.396761 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79f6755-8dab-4c54-b8e1-db5a5e032726-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g\" (UID: \"a79f6755-8dab-4c54-b8e1-db5a5e032726\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:02.397764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.397745 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79f6755-8dab-4c54-b8e1-db5a5e032726-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g\" (UID: \"a79f6755-8dab-4c54-b8e1-db5a5e032726\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:02.407650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.407626 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:02.454337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.454308 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26jpc\" (UniqueName: \"kubernetes.io/projected/a79f6755-8dab-4c54-b8e1-db5a5e032726-kube-api-access-26jpc\") pod \"payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g\" (UID: \"a79f6755-8dab-4c54-b8e1-db5a5e032726\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:02.566591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.566556 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:02.782295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.782267 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv"] Oct 30 21:29:02.785414 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:02.785388 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0a50aa_2cca_40ab_8ca0_794f5b0dcf85.slice/crio-8d87b98405c1c0d70eb82033c2a34fc34383623090d4f78d9a1cf6acda1dfc07 WatchSource:0}: Error finding container 8d87b98405c1c0d70eb82033c2a34fc34383623090d4f78d9a1cf6acda1dfc07: Status 404 returned error can't find the container with id 8d87b98405c1c0d70eb82033c2a34fc34383623090d4f78d9a1cf6acda1dfc07 Oct 30 21:29:02.941508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.941469 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g"] Oct 30 21:29:02.944132 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:02.944106 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79f6755_8dab_4c54_b8e1_db5a5e032726.slice/crio-16fc34d4e4cfe68a648bd926d83d4e922c3e956f4961ba4f0de88d1140d87a9b WatchSource:0}: Error finding container 16fc34d4e4cfe68a648bd926d83d4e922c3e956f4961ba4f0de88d1140d87a9b: Status 404 returned error can't find the container with id 16fc34d4e4cfe68a648bd926d83d4e922c3e956f4961ba4f0de88d1140d87a9b Oct 30 21:29:02.985974 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.985940 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" event={"ID":"a79f6755-8dab-4c54-b8e1-db5a5e032726","Type":"ContainerStarted","Data":"16fc34d4e4cfe68a648bd926d83d4e922c3e956f4961ba4f0de88d1140d87a9b"} Oct 30 21:29:02.987394 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.987372 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" event={"ID":"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85","Type":"ContainerStarted","Data":"8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b"} Oct 30 21:29:02.987528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.987413 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" event={"ID":"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85","Type":"ContainerStarted","Data":"8d87b98405c1c0d70eb82033c2a34fc34383623090d4f78d9a1cf6acda1dfc07"} Oct 30 21:29:02.987587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:02.987525 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:03.102223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:03.102167 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" podStartSLOduration=1.102151414 podStartE2EDuration="1.102151414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:03.099338636 +0000 UTC m=+733.232711834" watchObservedRunningTime="2026-10-30 21:29:03.102151414 +0000 UTC m=+733.235524669" Oct 30 21:29:03.993558 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:03.993515 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" event={"ID":"a79f6755-8dab-4c54-b8e1-db5a5e032726","Type":"ContainerStarted","Data":"cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d"} Oct 30 21:29:03.993791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:03.993771 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:04.109533 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.109453 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" podStartSLOduration=2.109436329 podStartE2EDuration="2.109436329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:04.1074028 +0000 UTC m=+734.240776000" watchObservedRunningTime="2026-10-30 21:29:04.109436329 +0000 UTC m=+734.242809526" Oct 30 21:29:04.560890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.560857 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d"] Oct 30 21:29:04.564532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.564483 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:04.619780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.619744 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e4dadd\"" Oct 30 21:29:04.622019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.621991 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d"] Oct 30 21:29:04.622538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.622515 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d\" (UID: \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:04.622625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.622569 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nrbk\" (UniqueName: \"kubernetes.io/projected/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-kube-api-access-8nrbk\") pod \"payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d\" (UID: \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:04.669725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.669687 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e4dadd-dockercfg-ndrpp\"" Oct 30 21:29:04.711028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.711001 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs"] Oct 30 21:29:04.714477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.714460 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:04.723506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.723463 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d\" (UID: \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:04.723634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.723538 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nrbk\" (UniqueName: \"kubernetes.io/projected/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-kube-api-access-8nrbk\") pod \"payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d\" (UID: \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:04.724209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.724187 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d\" (UID: \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:04.771878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.771849 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs"] Oct 30 21:29:04.780624 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.780599 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nrbk\" (UniqueName: \"kubernetes.io/projected/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-kube-api-access-8nrbk\") pod \"payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d\" (UID: \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:04.824762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.824689 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb576fcd-2113-4ce7-92b7-852040b4aa1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs\" (UID: \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:04.824762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.824723 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdk9b\" (UniqueName: \"kubernetes.io/projected/fb576fcd-2113-4ce7-92b7-852040b4aa1e-kube-api-access-sdk9b\") pod \"payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs\" (UID: \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:04.877760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.877725 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:04.927280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.927237 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb576fcd-2113-4ce7-92b7-852040b4aa1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs\" (UID: \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:04.927418 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.927292 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdk9b\" (UniqueName: \"kubernetes.io/projected/fb576fcd-2113-4ce7-92b7-852040b4aa1e-kube-api-access-sdk9b\") pod \"payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs\" (UID: \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:04.943970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.940407 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb576fcd-2113-4ce7-92b7-852040b4aa1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs\" (UID: \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:04.996421 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:04.996390 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdk9b\" (UniqueName: \"kubernetes.io/projected/fb576fcd-2113-4ce7-92b7-852040b4aa1e-kube-api-access-sdk9b\") pod \"payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs\" (UID: \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:05.027580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:05.027548 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:05.246057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:05.246025 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d"] Oct 30 21:29:05.249401 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:05.249376 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1df4872_bfd8_40a7_9cf1_ca7acf1216f2.slice/crio-a35dde6815d98535bfa0a70b08fb584cee26f5e47e6019e1024e4005633bf7b9 WatchSource:0}: Error finding container a35dde6815d98535bfa0a70b08fb584cee26f5e47e6019e1024e4005633bf7b9: Status 404 returned error can't find the container with id a35dde6815d98535bfa0a70b08fb584cee26f5e47e6019e1024e4005633bf7b9 Oct 30 21:29:05.398817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:05.398792 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs"] Oct 30 21:29:05.401220 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:05.401190 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb576fcd_2113_4ce7_92b7_852040b4aa1e.slice/crio-9ae0de1fb0d53092d8dfa68c419780e62c3c071082e5b80bfe7294675d7f9d14 WatchSource:0}: Error finding container 9ae0de1fb0d53092d8dfa68c419780e62c3c071082e5b80bfe7294675d7f9d14: Status 404 returned error can't find the container with id 9ae0de1fb0d53092d8dfa68c419780e62c3c071082e5b80bfe7294675d7f9d14 Oct 30 21:29:06.003699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:06.003667 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" event={"ID":"fb576fcd-2113-4ce7-92b7-852040b4aa1e","Type":"ContainerStarted","Data":"bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9"} Oct 30 21:29:06.003899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:06.003715 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" event={"ID":"fb576fcd-2113-4ce7-92b7-852040b4aa1e","Type":"ContainerStarted","Data":"9ae0de1fb0d53092d8dfa68c419780e62c3c071082e5b80bfe7294675d7f9d14"} Oct 30 21:29:06.003899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:06.003755 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:06.005340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:06.005309 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" event={"ID":"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2","Type":"ContainerStarted","Data":"b31041897174bef0f9f069e55cf9f9f388c6f26e5ed1b930018941d348fd012d"} Oct 30 21:29:06.005454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:06.005354 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" event={"ID":"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2","Type":"ContainerStarted","Data":"a35dde6815d98535bfa0a70b08fb584cee26f5e47e6019e1024e4005633bf7b9"} Oct 30 21:29:06.005454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:06.005435 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:06.127645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:06.127587 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" podStartSLOduration=2.127569849 podStartE2EDuration="2.127569849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:06.124480514 +0000 UTC m=+736.257853715" watchObservedRunningTime="2026-10-30 21:29:06.127569849 +0000 UTC m=+736.260943066" Oct 30 21:29:06.242512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:06.241386 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" podStartSLOduration=2.241365529 podStartE2EDuration="2.241365529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:06.238464338 +0000 UTC m=+736.371837536" watchObservedRunningTime="2026-10-30 21:29:06.241365529 +0000 UTC m=+736.374738730" Oct 30 21:29:07.088821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.088790 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984"] Oct 30 21:29:07.092369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.092353 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:07.153413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.153377 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvpd\" (UniqueName: \"kubernetes.io/projected/6b9e0bc4-bf67-413f-a580-974404f77720-kube-api-access-7dvpd\") pod \"payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984\" (UID: \"6b9e0bc4-bf67-413f-a580-974404f77720\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:07.153644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.153450 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e0bc4-bf67-413f-a580-974404f77720-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984\" (UID: \"6b9e0bc4-bf67-413f-a580-974404f77720\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:07.198006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.196230 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-07fa8e-dockercfg-kxd8f\"" Oct 30 21:29:07.198006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.196317 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-07fa8e\"" Oct 30 21:29:07.199729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.199700 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984"] Oct 30 21:29:07.247690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.247659 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9"] Oct 30 21:29:07.251295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.251278 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:07.254623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.254601 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvpd\" (UniqueName: \"kubernetes.io/projected/6b9e0bc4-bf67-413f-a580-974404f77720-kube-api-access-7dvpd\") pod \"payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984\" (UID: \"6b9e0bc4-bf67-413f-a580-974404f77720\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:07.254702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.254685 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e0bc4-bf67-413f-a580-974404f77720-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984\" (UID: \"6b9e0bc4-bf67-413f-a580-974404f77720\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:07.255510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.255465 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e0bc4-bf67-413f-a580-974404f77720-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984\" (UID: \"6b9e0bc4-bf67-413f-a580-974404f77720\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:07.288746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.288715 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9"] Oct 30 21:29:07.310755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.310723 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvpd\" (UniqueName: \"kubernetes.io/projected/6b9e0bc4-bf67-413f-a580-974404f77720-kube-api-access-7dvpd\") pod \"payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984\" (UID: \"6b9e0bc4-bf67-413f-a580-974404f77720\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:07.355524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.355419 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wv5w\" (UniqueName: \"kubernetes.io/projected/79acc862-069d-4c9e-9759-535067b6e745-kube-api-access-8wv5w\") pod \"payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9\" (UID: \"79acc862-069d-4c9e-9759-535067b6e745\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:07.355666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.355594 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acc862-069d-4c9e-9759-535067b6e745-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9\" (UID: \"79acc862-069d-4c9e-9759-535067b6e745\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:07.406259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.406221 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:07.456608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.456574 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wv5w\" (UniqueName: \"kubernetes.io/projected/79acc862-069d-4c9e-9759-535067b6e745-kube-api-access-8wv5w\") pod \"payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9\" (UID: \"79acc862-069d-4c9e-9759-535067b6e745\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:07.456774 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.456687 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acc862-069d-4c9e-9759-535067b6e745-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9\" (UID: \"79acc862-069d-4c9e-9759-535067b6e745\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:07.457373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.457353 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acc862-069d-4c9e-9759-535067b6e745-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9\" (UID: \"79acc862-069d-4c9e-9759-535067b6e745\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:07.512882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.512849 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wv5w\" (UniqueName: \"kubernetes.io/projected/79acc862-069d-4c9e-9759-535067b6e745-kube-api-access-8wv5w\") pod \"payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9\" (UID: \"79acc862-069d-4c9e-9759-535067b6e745\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:07.564651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.564617 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:07.786779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.786721 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984"] Oct 30 21:29:07.789052 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:07.789027 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9e0bc4_bf67_413f_a580_974404f77720.slice/crio-33ba5438cf2795649d90b47c0c30ac96dd702bbc40adab000828c6c3ecc157ff WatchSource:0}: Error finding container 33ba5438cf2795649d90b47c0c30ac96dd702bbc40adab000828c6c3ecc157ff: Status 404 returned error can't find the container with id 33ba5438cf2795649d90b47c0c30ac96dd702bbc40adab000828c6c3ecc157ff Oct 30 21:29:07.938286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:07.938257 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9"] Oct 30 21:29:07.940890 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:07.940862 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79acc862_069d_4c9e_9759_535067b6e745.slice/crio-fc7db092ee0ff66c8efc46463bb76e6d0d507d9ce9121ef492222a3e45183f1d WatchSource:0}: Error finding container fc7db092ee0ff66c8efc46463bb76e6d0d507d9ce9121ef492222a3e45183f1d: Status 404 returned error can't find the container with id fc7db092ee0ff66c8efc46463bb76e6d0d507d9ce9121ef492222a3e45183f1d Oct 30 21:29:08.016740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:08.016707 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" event={"ID":"6b9e0bc4-bf67-413f-a580-974404f77720","Type":"ContainerStarted","Data":"6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e"} Oct 30 21:29:08.016906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:08.016759 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" event={"ID":"6b9e0bc4-bf67-413f-a580-974404f77720","Type":"ContainerStarted","Data":"33ba5438cf2795649d90b47c0c30ac96dd702bbc40adab000828c6c3ecc157ff"} Oct 30 21:29:08.016906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:08.016887 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:08.018401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:08.018377 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" event={"ID":"79acc862-069d-4c9e-9759-535067b6e745","Type":"ContainerStarted","Data":"398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6"} Oct 30 21:29:08.018529 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:08.018417 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" event={"ID":"79acc862-069d-4c9e-9759-535067b6e745","Type":"ContainerStarted","Data":"fc7db092ee0ff66c8efc46463bb76e6d0d507d9ce9121ef492222a3e45183f1d"} Oct 30 21:29:08.018529 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:08.018453 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:08.136259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:08.136203 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" podStartSLOduration=1.136188935 podStartE2EDuration="1.136188935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:08.134685228 +0000 UTC m=+738.268058426" watchObservedRunningTime="2026-10-30 21:29:08.136188935 +0000 UTC m=+738.269562133" Oct 30 21:29:08.251309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:08.251183 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" podStartSLOduration=1.25116694 podStartE2EDuration="1.25116694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:08.248200534 +0000 UTC m=+738.381573733" watchObservedRunningTime="2026-10-30 21:29:08.25116694 +0000 UTC m=+738.384540139" Oct 30 21:29:11.903887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:11.903852 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq"] Oct 30 21:29:11.907575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:11.907558 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:11.965803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:11.965734 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-446ca2\"" Oct 30 21:29:12.009557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.009522 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfpw\" (UniqueName: \"kubernetes.io/projected/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-kube-api-access-4cfpw\") pod \"payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq\" (UID: \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:12.009755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.009667 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq\" (UID: \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:12.010669 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.010651 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-446ca2-dockercfg-j7q28\"" Oct 30 21:29:12.012987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.012958 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq"] Oct 30 21:29:12.024201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.024169 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc"] Oct 30 21:29:12.052847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.052820 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:12.093601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.093567 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc"] Oct 30 21:29:12.110264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.110237 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af111955-96d6-46e2-8103-2e4f51b8a622-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc\" (UID: \"af111955-96d6-46e2-8103-2e4f51b8a622\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:12.110428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.110276 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8x59\" (UniqueName: \"kubernetes.io/projected/af111955-96d6-46e2-8103-2e4f51b8a622-kube-api-access-l8x59\") pod \"payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc\" (UID: \"af111955-96d6-46e2-8103-2e4f51b8a622\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:12.110428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.110335 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfpw\" (UniqueName: \"kubernetes.io/projected/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-kube-api-access-4cfpw\") pod \"payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq\" (UID: \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:12.110524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.110458 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq\" (UID: \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:12.111175 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.111156 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq\" (UID: \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:12.185393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.185313 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfpw\" (UniqueName: \"kubernetes.io/projected/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-kube-api-access-4cfpw\") pod \"payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq\" (UID: \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:12.211343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.211312 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af111955-96d6-46e2-8103-2e4f51b8a622-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc\" (UID: \"af111955-96d6-46e2-8103-2e4f51b8a622\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:12.211562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.211357 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8x59\" (UniqueName: \"kubernetes.io/projected/af111955-96d6-46e2-8103-2e4f51b8a622-kube-api-access-l8x59\") pod \"payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc\" (UID: \"af111955-96d6-46e2-8103-2e4f51b8a622\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:12.212034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.212015 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af111955-96d6-46e2-8103-2e4f51b8a622-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc\" (UID: \"af111955-96d6-46e2-8103-2e4f51b8a622\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:12.221273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.221251 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:12.268892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.268859 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8x59\" (UniqueName: \"kubernetes.io/projected/af111955-96d6-46e2-8103-2e4f51b8a622-kube-api-access-l8x59\") pod \"payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc\" (UID: \"af111955-96d6-46e2-8103-2e4f51b8a622\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:12.376512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.375511 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:12.482652 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:29:12.482569 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:29:12.600312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.600284 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq"] Oct 30 21:29:12.602811 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:12.602771 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dfcdda_bbd5_4ebb_9829_f6c23e06982f.slice/crio-3ea485008fc99e34ff5824f205b7322eeddb8965ecd48a150684a8e5916a81e5 WatchSource:0}: Error finding container 3ea485008fc99e34ff5824f205b7322eeddb8965ecd48a150684a8e5916a81e5: Status 404 returned error can't find the container with id 3ea485008fc99e34ff5824f205b7322eeddb8965ecd48a150684a8e5916a81e5 Oct 30 21:29:12.753871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:12.753844 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc"] Oct 30 21:29:12.756021 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:12.755993 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf111955_96d6_46e2_8103_2e4f51b8a622.slice/crio-5dc3393b4fc5ef5e11b8c249e9f893f25d4acd728cf25e77f89212ee6031d10b WatchSource:0}: Error finding container 5dc3393b4fc5ef5e11b8c249e9f893f25d4acd728cf25e77f89212ee6031d10b: Status 404 returned error can't find the container with id 5dc3393b4fc5ef5e11b8c249e9f893f25d4acd728cf25e77f89212ee6031d10b Oct 30 21:29:13.062616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.062564 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" event={"ID":"af111955-96d6-46e2-8103-2e4f51b8a622","Type":"ContainerStarted","Data":"d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da"} Oct 30 21:29:13.063044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.062666 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" event={"ID":"af111955-96d6-46e2-8103-2e4f51b8a622","Type":"ContainerStarted","Data":"5dc3393b4fc5ef5e11b8c249e9f893f25d4acd728cf25e77f89212ee6031d10b"} Oct 30 21:29:13.063044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.062696 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:13.064228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.064200 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" event={"ID":"06dfcdda-bbd5-4ebb-9829-f6c23e06982f","Type":"ContainerStarted","Data":"59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0"} Oct 30 21:29:13.064317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.064250 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" event={"ID":"06dfcdda-bbd5-4ebb-9829-f6c23e06982f","Type":"ContainerStarted","Data":"3ea485008fc99e34ff5824f205b7322eeddb8965ecd48a150684a8e5916a81e5"} Oct 30 21:29:13.064317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.064270 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:13.177685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.177544 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" podStartSLOduration=2.177524926 podStartE2EDuration="2.177524926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:13.176354105 +0000 UTC m=+743.309727302" watchObservedRunningTime="2026-10-30 21:29:13.177524926 +0000 UTC m=+743.310898125" Oct 30 21:29:13.292037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.291978 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" podStartSLOduration=2.291959578 podStartE2EDuration="2.291959578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:13.289870475 +0000 UTC m=+743.423243672" watchObservedRunningTime="2026-10-30 21:29:13.291959578 +0000 UTC m=+743.425332777" Oct 30 21:29:13.996698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:13.996666 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:29:15.001984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:15.001951 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:29:17.013996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:17.013965 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:29:17.014408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:17.014106 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:29:18.993663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:18.993627 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd"] Oct 30 21:29:18.998425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:18.998408 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:19.027589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.027560 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 21:29:19.027727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.027630 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 21:29:19.053976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.053947 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd"] Oct 30 21:29:19.086478 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.086288 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9nm\" (UniqueName: \"kubernetes.io/projected/9611497d-715d-438f-98c1-d6b4332ef005-kube-api-access-vs9nm\") pod \"payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd\" (UID: \"9611497d-715d-438f-98c1-d6b4332ef005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:19.086478 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.086372 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9611497d-715d-438f-98c1-d6b4332ef005-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd\" (UID: \"9611497d-715d-438f-98c1-d6b4332ef005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:19.101179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.101141 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-89c0fd\"" Oct 30 21:29:19.105200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.105172 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-89c0fd-dockercfg-pdszx\"" Oct 30 21:29:19.133854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.133816 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt"] Oct 30 21:29:19.137640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.137623 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:19.187706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.187678 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9nm\" (UniqueName: \"kubernetes.io/projected/9611497d-715d-438f-98c1-d6b4332ef005-kube-api-access-vs9nm\") pod \"payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd\" (UID: \"9611497d-715d-438f-98c1-d6b4332ef005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:19.187900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.187752 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9611497d-715d-438f-98c1-d6b4332ef005-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd\" (UID: \"9611497d-715d-438f-98c1-d6b4332ef005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:19.188450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.188431 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9611497d-715d-438f-98c1-d6b4332ef005-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd\" (UID: \"9611497d-715d-438f-98c1-d6b4332ef005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:19.194894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.194870 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt"] Oct 30 21:29:19.244341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.244263 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9nm\" (UniqueName: \"kubernetes.io/projected/9611497d-715d-438f-98c1-d6b4332ef005-kube-api-access-vs9nm\") pod \"payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd\" (UID: \"9611497d-715d-438f-98c1-d6b4332ef005\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:19.289320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.289287 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rn5\" (UniqueName: \"kubernetes.io/projected/eb8f479d-3076-4853-b172-8a11f4721edd-kube-api-access-n5rn5\") pod \"payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt\" (UID: \"eb8f479d-3076-4853-b172-8a11f4721edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:19.289509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.289415 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f479d-3076-4853-b172-8a11f4721edd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt\" (UID: \"eb8f479d-3076-4853-b172-8a11f4721edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:19.312173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.312142 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:19.399076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.399041 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rn5\" (UniqueName: \"kubernetes.io/projected/eb8f479d-3076-4853-b172-8a11f4721edd-kube-api-access-n5rn5\") pod \"payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt\" (UID: \"eb8f479d-3076-4853-b172-8a11f4721edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:19.399291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.399110 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f479d-3076-4853-b172-8a11f4721edd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt\" (UID: \"eb8f479d-3076-4853-b172-8a11f4721edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:19.400011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.399983 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f479d-3076-4853-b172-8a11f4721edd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt\" (UID: \"eb8f479d-3076-4853-b172-8a11f4721edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:19.455842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.455812 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rn5\" (UniqueName: \"kubernetes.io/projected/eb8f479d-3076-4853-b172-8a11f4721edd-kube-api-access-n5rn5\") pod \"payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt\" (UID: \"eb8f479d-3076-4853-b172-8a11f4721edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:19.707399 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.707379 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd"] Oct 30 21:29:19.709386 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:19.709357 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9611497d_715d_438f_98c1_d6b4332ef005.slice/crio-0b5d0957f3e3ca09926a45af224db86772554a7284f9f13ea91e542151aa45fa WatchSource:0}: Error finding container 0b5d0957f3e3ca09926a45af224db86772554a7284f9f13ea91e542151aa45fa: Status 404 returned error can't find the container with id 0b5d0957f3e3ca09926a45af224db86772554a7284f9f13ea91e542151aa45fa Oct 30 21:29:19.752252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:19.752228 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:20.112033 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:20.111998 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" event={"ID":"9611497d-715d-438f-98c1-d6b4332ef005","Type":"ContainerStarted","Data":"26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d"} Oct 30 21:29:20.112465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:20.112048 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" event={"ID":"9611497d-715d-438f-98c1-d6b4332ef005","Type":"ContainerStarted","Data":"0b5d0957f3e3ca09926a45af224db86772554a7284f9f13ea91e542151aa45fa"} Oct 30 21:29:20.112465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:20.112091 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:20.139382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:20.139357 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt"] Oct 30 21:29:20.141923 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:20.141899 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8f479d_3076_4853_b172_8a11f4721edd.slice/crio-4838062a167c09fb9ec674b702ef47e3663f3f7447c15b4a441d7d807dc6f30b WatchSource:0}: Error finding container 4838062a167c09fb9ec674b702ef47e3663f3f7447c15b4a441d7d807dc6f30b: Status 404 returned error can't find the container with id 4838062a167c09fb9ec674b702ef47e3663f3f7447c15b4a441d7d807dc6f30b Oct 30 21:29:20.224590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:20.224533 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" podStartSLOduration=2.224519357 podStartE2EDuration="2.224519357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:20.223817723 +0000 UTC m=+750.357190917" watchObservedRunningTime="2026-10-30 21:29:20.224519357 +0000 UTC m=+750.357892548" Oct 30 21:29:21.125079 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:21.125043 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" event={"ID":"eb8f479d-3076-4853-b172-8a11f4721edd","Type":"ContainerStarted","Data":"da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846"} Oct 30 21:29:21.125551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:21.125121 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" event={"ID":"eb8f479d-3076-4853-b172-8a11f4721edd","Type":"ContainerStarted","Data":"4838062a167c09fb9ec674b702ef47e3663f3f7447c15b4a441d7d807dc6f30b"} Oct 30 21:29:21.125551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:21.125142 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:21.239650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:21.239598 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" podStartSLOduration=2.239584525 podStartE2EDuration="2.239584525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:21.237905175 +0000 UTC m=+751.371278373" watchObservedRunningTime="2026-10-30 21:29:21.239584525 +0000 UTC m=+751.372957722" Oct 30 21:29:23.954569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:23.953653 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt"] Oct 30 21:29:23.959333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:23.959311 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:23.969535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:23.967932 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt"] Oct 30 21:29:24.046686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.036771 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1b5b5f\"" Oct 30 21:29:24.046686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.040258 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1b5b5f-dockercfg-5w9h4\"" Oct 30 21:29:24.065499 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.065445 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2pkt\" (UniqueName: \"kubernetes.io/projected/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-kube-api-access-n2pkt\") pod \"payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt\" (UID: \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:24.065719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.065697 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt\" (UID: \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:24.073639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.073615 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 21:29:24.073896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.073880 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 21:29:24.166452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.166412 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt\" (UID: \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:24.166452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.166450 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2pkt\" (UniqueName: \"kubernetes.io/projected/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-kube-api-access-n2pkt\") pod \"payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt\" (UID: \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:24.167292 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.167273 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt\" (UID: \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:24.251079 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.251000 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2pkt\" (UniqueName: \"kubernetes.io/projected/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-kube-api-access-n2pkt\") pod \"payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt\" (UID: \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:24.279246 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.278961 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:24.726516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:24.713538 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt"] Oct 30 21:29:24.753641 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:24.736177 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1c303c_e1ac_4e5e_bfd0_a66142a14770.slice/crio-28550eec84e710398a10e4442b02584bfe3430ac778c8553b12ecc8fb0094f88 WatchSource:0}: Error finding container 28550eec84e710398a10e4442b02584bfe3430ac778c8553b12ecc8fb0094f88: Status 404 returned error can't find the container with id 28550eec84e710398a10e4442b02584bfe3430ac778c8553b12ecc8fb0094f88 Oct 30 21:29:25.163172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:25.163124 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" event={"ID":"3d1c303c-e1ac-4e5e-bfd0-a66142a14770","Type":"ContainerStarted","Data":"9ffcba5d88db4c0c850b5d2e5ab0d314de891a6ac478cf9f0951c577a6e849ea"} Oct 30 21:29:25.163700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:25.163197 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" event={"ID":"3d1c303c-e1ac-4e5e-bfd0-a66142a14770","Type":"ContainerStarted","Data":"28550eec84e710398a10e4442b02584bfe3430ac778c8553b12ecc8fb0094f88"} Oct 30 21:29:25.163700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:25.163259 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:25.282776 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:25.282720 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" podStartSLOduration=2.282703436 podStartE2EDuration="2.282703436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:25.280322261 +0000 UTC m=+755.413695473" watchObservedRunningTime="2026-10-30 21:29:25.282703436 +0000 UTC m=+755.416076686" Oct 30 21:29:31.129034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:31.129006 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:29:32.134286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.134256 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:29:32.209265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.209228 4526 generic.go:356] "Generic (PLEG): container finished" podID="cda6d95a-c501-4387-b6c8-694af23150e6" containerID="7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c" exitCode=0 Oct 30 21:29:32.209265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.209268 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" event={"ID":"cda6d95a-c501-4387-b6c8-694af23150e6","Type":"ContainerDied","Data":"7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c"} Oct 30 21:29:32.209749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.209725 4526 scope.go:122] "RemoveContainer" containerID="7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c" Oct 30 21:29:32.211358 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.211332 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:32.211462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.211416 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:32.211550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.211469 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:32.211749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.211728 4526 generic.go:356] "Generic (PLEG): container finished" podID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerID="68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee" exitCode=0 Oct 30 21:29:32.211844 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.211794 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" event={"ID":"61876d79-62d0-49d1-9f3a-8e78d0e1e67b","Type":"ContainerDied","Data":"68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee"} Oct 30 21:29:32.212163 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.212150 4526 scope.go:122] "RemoveContainer" containerID="68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee" Oct 30 21:29:32.214094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.214069 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:32.214164 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.214150 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:32.214221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:32.214199 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:33.249570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:33.247548 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" event={"ID":"cda6d95a-c501-4387-b6c8-694af23150e6","Type":"ContainerStarted","Data":"eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66"} Oct 30 21:29:33.251720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:33.251687 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" event={"ID":"61876d79-62d0-49d1-9f3a-8e78d0e1e67b","Type":"ContainerStarted","Data":"472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b"} Oct 30 21:29:33.408680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:33.408162 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:29:33.408680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:33.408314 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:29:33.408680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:33.408367 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:29:34.068610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:34.068578 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:29:34.069697 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:34.069671 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:29:34.069815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:34.069726 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:29:34.405405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:34.404913 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:29:34.405405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:34.404974 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:29:35.069903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:35.069850 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:29:35.070128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:35.069912 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:29:35.406110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:35.406019 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:29:35.406742 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:35.406701 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:29:35.931647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:35.931604 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:29:35.932029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:35.932010 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:29:36.069450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:36.069414 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:29:36.069648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:36.069472 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:29:36.171500 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:36.171456 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:29:36.404067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:36.404031 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:29:36.427857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:36.427819 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:29:37.069366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:37.069336 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:29:37.069540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:37.069385 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:29:37.405999 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:37.405905 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:29:37.406274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:37.406244 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:29:38.069370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:38.069340 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:29:38.069754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:38.069389 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 21:29:38.408903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:38.408809 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:29:38.409099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:38.409029 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 21:29:39.073906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:39.073871 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:29:39.075265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:39.075243 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 21:29:39.407870 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:39.407796 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:29:39.408984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:39.408963 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 21:29:42.478452 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:29:42.478419 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:29:42.657458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.657431 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5"] Oct 30 21:29:42.662368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.662346 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.728911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.728822 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5"] Oct 30 21:29:42.757831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.757798 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-model-cache\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.757995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.757913 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-home\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.757995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.757933 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-dshm\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.758100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.758072 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzpv4\" (UniqueName: \"kubernetes.io/projected/ea8af78e-f20e-4709-8198-3d175d15bba3-kube-api-access-kzpv4\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.758214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.758162 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-tmp-dir\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.758279 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.758222 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8af78e-f20e-4709-8198-3d175d15bba3-tls-certs\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.775345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.775313 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9796f1\"/\"kube-root-ca.crt\"" Oct 30 21:29:42.775608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.775592 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9796f1\"/\"default-dockercfg-4fd4p\"" Oct 30 21:29:42.775682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.775611 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9796f1\"/\"openshift-service-ca.crt\"" Oct 30 21:29:42.775682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.775665 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9796f1\"/\"simulator-w3-9796f1-kserve-self-signed-certs\"" Oct 30 21:29:42.859758 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.859718 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzpv4\" (UniqueName: \"kubernetes.io/projected/ea8af78e-f20e-4709-8198-3d175d15bba3-kube-api-access-kzpv4\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.859937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.859777 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-tmp-dir\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.859937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.859835 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8af78e-f20e-4709-8198-3d175d15bba3-tls-certs\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.859937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.859867 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-model-cache\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.859937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.859920 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-home\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.859937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.859936 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-dshm\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.860249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.860233 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-tmp-dir\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.860319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.860302 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-model-cache\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.860376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.860334 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-home\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.862392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.862371 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-dshm\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.862569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.862544 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8af78e-f20e-4709-8198-3d175d15bba3-tls-certs\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.920059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.920021 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzpv4\" (UniqueName: \"kubernetes.io/projected/ea8af78e-f20e-4709-8198-3d175d15bba3-kube-api-access-kzpv4\") pod \"simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:42.974831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:42.974792 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:43.353445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:43.353413 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5"] Oct 30 21:29:43.355754 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:29:43.355722 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea8af78e_f20e_4709_8198_3d175d15bba3.slice/crio-e9e818cdce2f84f58e32be661291daf11720428f5547506b6359cddfbfbb17ef WatchSource:0}: Error finding container e9e818cdce2f84f58e32be661291daf11720428f5547506b6359cddfbfbb17ef: Status 404 returned error can't find the container with id e9e818cdce2f84f58e32be661291daf11720428f5547506b6359cddfbfbb17ef Oct 30 21:29:44.308055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:44.308017 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" event={"ID":"ea8af78e-f20e-4709-8198-3d175d15bba3","Type":"ContainerStarted","Data":"03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e"} Oct 30 21:29:44.308055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:44.308059 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" event={"ID":"ea8af78e-f20e-4709-8198-3d175d15bba3","Type":"ContainerStarted","Data":"e9e818cdce2f84f58e32be661291daf11720428f5547506b6359cddfbfbb17ef"} Oct 30 21:29:44.422332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:44.422270 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podStartSLOduration=2.422251686 podStartE2EDuration="2.422251686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:44.421215338 +0000 UTC m=+774.554588536" watchObservedRunningTime="2026-10-30 21:29:44.422251686 +0000 UTC m=+774.555624885" Oct 30 21:29:44.975629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:44.975596 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:52.975188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:52.975150 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:52.988458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:52.988428 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:29:53.366999 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:29:53.366969 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:30:00.166687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.166657 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz"] Oct 30 21:30:00.172455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.172428 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj"] Oct 30 21:30:00.172634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.172571 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:00.176324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.176303 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc"] Oct 30 21:30:00.176475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.176404 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:00.180179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.180159 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:00.217042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.217008 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn"] Oct 30 21:30:00.221345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.221325 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz"] Oct 30 21:30:00.221452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.221436 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:00.224366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.224341 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:30:00.224366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.224355 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:30:00.225190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.225175 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:00.260284 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.260238 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqtn\" (UniqueName: \"kubernetes.io/projected/e0025b80-00e8-4fe1-88fa-69429daa452e-kube-api-access-4hqtn\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz\" (UID: \"e0025b80-00e8-4fe1-88fa-69429daa452e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:00.260472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.260323 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0025b80-00e8-4fe1-88fa-69429daa452e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz\" (UID: \"e0025b80-00e8-4fe1-88fa-69429daa452e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:00.263288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.263263 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:30:00.269586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.267727 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz"] Oct 30 21:30:00.270650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.270623 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj"] Oct 30 21:30:00.277222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.277189 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn"] Oct 30 21:30:00.317748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.317708 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz"] Oct 30 21:30:00.335330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.335115 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc"] Oct 30 21:30:00.361879 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.361845 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0025b80-00e8-4fe1-88fa-69429daa452e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz\" (UID: \"e0025b80-00e8-4fe1-88fa-69429daa452e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:00.362087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.361889 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drw5n\" (UniqueName: \"kubernetes.io/projected/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-kube-api-access-drw5n\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc\" (UID: \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:00.362087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.361934 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d304b1-7215-493c-a773-26c638be6775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz\" (UID: \"00d304b1-7215-493c-a773-26c638be6775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:00.362087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.361975 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqk24\" (UniqueName: \"kubernetes.io/projected/04683c2f-82e5-4131-b6cf-7a60a5045565-kube-api-access-tqk24\") pod \"maas-api-key-cleanup-29889930-c9vfn\" (UID: \"04683c2f-82e5-4131-b6cf-7a60a5045565\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:00.362087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.362065 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qzw\" (UniqueName: \"kubernetes.io/projected/00d304b1-7215-493c-a773-26c638be6775-kube-api-access-z9qzw\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz\" (UID: \"00d304b1-7215-493c-a773-26c638be6775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:00.362321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.362156 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj\" (UID: \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:00.362321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.362201 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc\" (UID: \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:00.362321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.362233 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqtn\" (UniqueName: \"kubernetes.io/projected/e0025b80-00e8-4fe1-88fa-69429daa452e-kube-api-access-4hqtn\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz\" (UID: \"e0025b80-00e8-4fe1-88fa-69429daa452e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:00.362321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.362270 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxqt\" (UniqueName: \"kubernetes.io/projected/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-kube-api-access-rdxqt\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj\" (UID: \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:00.362321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.362293 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04683c2f-82e5-4131-b6cf-7a60a5045565-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-c9vfn\" (UID: \"04683c2f-82e5-4131-b6cf-7a60a5045565\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:00.362741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.362720 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0025b80-00e8-4fe1-88fa-69429daa452e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz\" (UID: \"e0025b80-00e8-4fe1-88fa-69429daa452e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:00.418937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.418856 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqtn\" (UniqueName: \"kubernetes.io/projected/e0025b80-00e8-4fe1-88fa-69429daa452e-kube-api-access-4hqtn\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz\" (UID: \"e0025b80-00e8-4fe1-88fa-69429daa452e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:00.463902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.463862 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxqt\" (UniqueName: \"kubernetes.io/projected/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-kube-api-access-rdxqt\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj\" (UID: \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:00.463902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.463908 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04683c2f-82e5-4131-b6cf-7a60a5045565-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-c9vfn\" (UID: \"04683c2f-82e5-4131-b6cf-7a60a5045565\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:00.464172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.464084 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drw5n\" (UniqueName: \"kubernetes.io/projected/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-kube-api-access-drw5n\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc\" (UID: \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:00.464172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.464150 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d304b1-7215-493c-a773-26c638be6775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz\" (UID: \"00d304b1-7215-493c-a773-26c638be6775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:00.464297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.464201 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqk24\" (UniqueName: \"kubernetes.io/projected/04683c2f-82e5-4131-b6cf-7a60a5045565-kube-api-access-tqk24\") pod \"maas-api-key-cleanup-29889930-c9vfn\" (UID: \"04683c2f-82e5-4131-b6cf-7a60a5045565\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:00.464297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.464226 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qzw\" (UniqueName: \"kubernetes.io/projected/00d304b1-7215-493c-a773-26c638be6775-kube-api-access-z9qzw\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz\" (UID: \"00d304b1-7215-493c-a773-26c638be6775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:00.464297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.464266 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj\" (UID: \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:00.464449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.464315 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc\" (UID: \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:00.464713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.464683 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04683c2f-82e5-4131-b6cf-7a60a5045565-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-c9vfn\" (UID: \"04683c2f-82e5-4131-b6cf-7a60a5045565\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:00.464967 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.464948 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d304b1-7215-493c-a773-26c638be6775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz\" (UID: \"00d304b1-7215-493c-a773-26c638be6775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:00.465124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.465103 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc\" (UID: \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:00.465162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.465104 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj\" (UID: \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:00.486867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.486842 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:00.520807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.520774 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxqt\" (UniqueName: \"kubernetes.io/projected/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-kube-api-access-rdxqt\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj\" (UID: \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:00.521074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.521035 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqk24\" (UniqueName: \"kubernetes.io/projected/04683c2f-82e5-4131-b6cf-7a60a5045565-kube-api-access-tqk24\") pod \"maas-api-key-cleanup-29889930-c9vfn\" (UID: \"04683c2f-82e5-4131-b6cf-7a60a5045565\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:00.521160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.521145 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drw5n\" (UniqueName: \"kubernetes.io/projected/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-kube-api-access-drw5n\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc\" (UID: \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:00.521218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.521208 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qzw\" (UniqueName: \"kubernetes.io/projected/00d304b1-7215-493c-a773-26c638be6775-kube-api-access-z9qzw\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz\" (UID: \"00d304b1-7215-493c-a773-26c638be6775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:00.539104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.539068 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:00.544989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.544965 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:00.795503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.795454 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:00.803755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.803720 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:00.894298 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:00.894265 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz"] Oct 30 21:30:00.906787 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:30:00.906752 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0025b80_00e8_4fe1_88fa_69429daa452e.slice/crio-31efad9b9617e72aba2f0a061f8a9716c17cb62a10d976eba36129436cf67ebc WatchSource:0}: Error finding container 31efad9b9617e72aba2f0a061f8a9716c17cb62a10d976eba36129436cf67ebc: Status 404 returned error can't find the container with id 31efad9b9617e72aba2f0a061f8a9716c17cb62a10d976eba36129436cf67ebc Oct 30 21:30:01.018816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.018663 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn"] Oct 30 21:30:01.030237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.030203 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz"] Oct 30 21:30:01.032619 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:30:01.032233 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d304b1_7215_493c_a773_26c638be6775.slice/crio-5b874a087cbe84b67bf44128374cbba5043359ca9cf0efb622ad2f794f3c427b WatchSource:0}: Error finding container 5b874a087cbe84b67bf44128374cbba5043359ca9cf0efb622ad2f794f3c427b: Status 404 returned error can't find the container with id 5b874a087cbe84b67bf44128374cbba5043359ca9cf0efb622ad2f794f3c427b Oct 30 21:30:01.035913 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:30:01.035819 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04683c2f_82e5_4131_b6cf_7a60a5045565.slice/crio-58889ffaa029e767ff70167c0f6422ddcda6a9e4ea20c2cecd68748b46bd5d6c WatchSource:0}: Error finding container 58889ffaa029e767ff70167c0f6422ddcda6a9e4ea20c2cecd68748b46bd5d6c: Status 404 returned error can't find the container with id 58889ffaa029e767ff70167c0f6422ddcda6a9e4ea20c2cecd68748b46bd5d6c Oct 30 21:30:01.216113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.216086 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj"] Oct 30 21:30:01.217886 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:30:01.217861 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b73e00e_1d35_4e34_ab57_fc9ab9e17c55.slice/crio-f43cd00af5089a0ac93f033a1b68aff9fcbba3cd3f1f0599781b930c7d085751 WatchSource:0}: Error finding container f43cd00af5089a0ac93f033a1b68aff9fcbba3cd3f1f0599781b930c7d085751: Status 404 returned error can't find the container with id f43cd00af5089a0ac93f033a1b68aff9fcbba3cd3f1f0599781b930c7d085751 Oct 30 21:30:01.223811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.223717 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc"] Oct 30 21:30:01.226459 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:30:01.226436 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3008baf_ffa9_41d9_bd5d_465ebfdbb813.slice/crio-0ac5a39358355b55c55dc570babc16a9f073fa6cc305a00b4e2159ffbd6f2d85 WatchSource:0}: Error finding container 0ac5a39358355b55c55dc570babc16a9f073fa6cc305a00b4e2159ffbd6f2d85: Status 404 returned error can't find the container with id 0ac5a39358355b55c55dc570babc16a9f073fa6cc305a00b4e2159ffbd6f2d85 Oct 30 21:30:01.398846 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.398799 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" event={"ID":"00d304b1-7215-493c-a773-26c638be6775","Type":"ContainerStarted","Data":"5b874a087cbe84b67bf44128374cbba5043359ca9cf0efb622ad2f794f3c427b"} Oct 30 21:30:01.399903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.399879 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" event={"ID":"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55","Type":"ContainerStarted","Data":"f43cd00af5089a0ac93f033a1b68aff9fcbba3cd3f1f0599781b930c7d085751"} Oct 30 21:30:01.400991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.400967 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" event={"ID":"f3008baf-ffa9-41d9-bd5d-465ebfdbb813","Type":"ContainerStarted","Data":"0ac5a39358355b55c55dc570babc16a9f073fa6cc305a00b4e2159ffbd6f2d85"} Oct 30 21:30:01.402110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.402087 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" event={"ID":"04683c2f-82e5-4131-b6cf-7a60a5045565","Type":"ContainerStarted","Data":"58889ffaa029e767ff70167c0f6422ddcda6a9e4ea20c2cecd68748b46bd5d6c"} Oct 30 21:30:01.403172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:01.403153 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" event={"ID":"e0025b80-00e8-4fe1-88fa-69429daa452e","Type":"ContainerStarted","Data":"31efad9b9617e72aba2f0a061f8a9716c17cb62a10d976eba36129436cf67ebc"} Oct 30 21:30:07.493234 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.493199 4526 generic.go:356] "Generic (PLEG): container finished" podID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerID="53c749ce824ad6820a4762d1d8323f7a60c91a2ce48d639b3979f900df77bcff" exitCode=60 Oct 30 21:30:07.497969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.493471 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" event={"ID":"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55","Type":"ContainerDied","Data":"53c749ce824ad6820a4762d1d8323f7a60c91a2ce48d639b3979f900df77bcff"} Oct 30 21:30:07.497969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.494159 4526 scope.go:122] "RemoveContainer" containerID="53c749ce824ad6820a4762d1d8323f7a60c91a2ce48d639b3979f900df77bcff" Oct 30 21:30:07.505032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.505003 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" event={"ID":"f3008baf-ffa9-41d9-bd5d-465ebfdbb813","Type":"ContainerStarted","Data":"f4ae4691a6b0f55921afc6af474858f1b470a1b7c7b242121d39071cf489f381"} Oct 30 21:30:07.508441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.508416 4526 generic.go:356] "Generic (PLEG): container finished" podID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerID="76dc0663881f80c371b646287208f77cc385e586129c1462d027d6ea9bf9c770" exitCode=60 Oct 30 21:30:07.508586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.508525 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" event={"ID":"04683c2f-82e5-4131-b6cf-7a60a5045565","Type":"ContainerDied","Data":"76dc0663881f80c371b646287208f77cc385e586129c1462d027d6ea9bf9c770"} Oct 30 21:30:07.508749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.508732 4526 scope.go:122] "RemoveContainer" containerID="76dc0663881f80c371b646287208f77cc385e586129c1462d027d6ea9bf9c770" Oct 30 21:30:07.510341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.510312 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" event={"ID":"e0025b80-00e8-4fe1-88fa-69429daa452e","Type":"ContainerStarted","Data":"eb807b0cbf41b4de8d89c35e273f6be94c06e76b482faa5df89a00ec1c2c0f38"} Oct 30 21:30:07.512037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.512015 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" event={"ID":"00d304b1-7215-493c-a773-26c638be6775","Type":"ContainerStarted","Data":"7d47a8b678828107f63e7fd0a5d442f6f7a1413c9d7da2c870f16050cf435496"} Oct 30 21:30:07.724977 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.724929 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" podStartSLOduration=2.328137548 podStartE2EDuration="7.724912387s" totalImagesPullingTime="5.396774839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:07.722997061 +0000 UTC m=+797.856370258" watchObservedRunningTime="2026-10-30 21:30:07.724912387 +0000 UTC m=+797.858285584" Oct 30 21:30:07.986712 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.986607 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" podStartSLOduration=2.394743123 podStartE2EDuration="7.986592466s" totalImagesPullingTime="5.591849343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:07.984970087 +0000 UTC m=+798.118343285" watchObservedRunningTime="2026-10-30 21:30:07.986592466 +0000 UTC m=+798.119965696" Oct 30 21:30:07.986994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:07.986973 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" podStartSLOduration=2.261896988 podStartE2EDuration="7.986965678s" totalImagesPullingTime="5.72506869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:07.855809546 +0000 UTC m=+797.989182745" watchObservedRunningTime="2026-10-30 21:30:07.986965678 +0000 UTC m=+798.120338876" Oct 30 21:30:08.518918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.518886 4526 generic.go:356] "Generic (PLEG): container finished" podID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerID="32849177e57ef7c27179169b7f3ed93e311b712ca430fb4caaef57241bd945f5" exitCode=60 Oct 30 21:30:08.519350 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.518964 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" event={"ID":"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55","Type":"ContainerDied","Data":"32849177e57ef7c27179169b7f3ed93e311b712ca430fb4caaef57241bd945f5"} Oct 30 21:30:08.519350 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.519021 4526 scope.go:122] "RemoveContainer" containerID="53c749ce824ad6820a4762d1d8323f7a60c91a2ce48d639b3979f900df77bcff" Oct 30 21:30:08.519350 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.519262 4526 scope.go:122] "RemoveContainer" containerID="32849177e57ef7c27179169b7f3ed93e311b712ca430fb4caaef57241bd945f5" Oct 30 21:30:08.519722 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:08.519695 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj_odh-ai-gateway-infra(3b73e00e-1d35-4e34-ab57-fc9ab9e17c55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" Oct 30 21:30:08.520857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.520835 4526 generic.go:356] "Generic (PLEG): container finished" podID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerID="f4ae4691a6b0f55921afc6af474858f1b470a1b7c7b242121d39071cf489f381" exitCode=60 Oct 30 21:30:08.520946 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.520920 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" event={"ID":"f3008baf-ffa9-41d9-bd5d-465ebfdbb813","Type":"ContainerDied","Data":"f4ae4691a6b0f55921afc6af474858f1b470a1b7c7b242121d39071cf489f381"} Oct 30 21:30:08.521116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.521101 4526 scope.go:122] "RemoveContainer" containerID="f4ae4691a6b0f55921afc6af474858f1b470a1b7c7b242121d39071cf489f381" Oct 30 21:30:08.522840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.522818 4526 generic.go:356] "Generic (PLEG): container finished" podID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerID="3a4d2deb50ca493fe09de2eabfdd107e1da38c2d3550b8fbbec10c34e2d3d44c" exitCode=60 Oct 30 21:30:08.522926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.522898 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" event={"ID":"04683c2f-82e5-4131-b6cf-7a60a5045565","Type":"ContainerDied","Data":"3a4d2deb50ca493fe09de2eabfdd107e1da38c2d3550b8fbbec10c34e2d3d44c"} Oct 30 21:30:08.523155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.523139 4526 scope.go:122] "RemoveContainer" containerID="3a4d2deb50ca493fe09de2eabfdd107e1da38c2d3550b8fbbec10c34e2d3d44c" Oct 30 21:30:08.523373 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:08.523353 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-c9vfn_odh-ai-gateway-infra(04683c2f-82e5-4131-b6cf-7a60a5045565)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" Oct 30 21:30:08.524561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.524539 4526 generic.go:356] "Generic (PLEG): container finished" podID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerID="eb807b0cbf41b4de8d89c35e273f6be94c06e76b482faa5df89a00ec1c2c0f38" exitCode=60 Oct 30 21:30:08.524650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.524624 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" event={"ID":"e0025b80-00e8-4fe1-88fa-69429daa452e","Type":"ContainerDied","Data":"eb807b0cbf41b4de8d89c35e273f6be94c06e76b482faa5df89a00ec1c2c0f38"} Oct 30 21:30:08.524868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.524851 4526 scope.go:122] "RemoveContainer" containerID="eb807b0cbf41b4de8d89c35e273f6be94c06e76b482faa5df89a00ec1c2c0f38" Oct 30 21:30:08.526332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.526313 4526 generic.go:356] "Generic (PLEG): container finished" podID="00d304b1-7215-493c-a773-26c638be6775" containerID="7d47a8b678828107f63e7fd0a5d442f6f7a1413c9d7da2c870f16050cf435496" exitCode=60 Oct 30 21:30:08.526404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.526344 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" event={"ID":"00d304b1-7215-493c-a773-26c638be6775","Type":"ContainerDied","Data":"7d47a8b678828107f63e7fd0a5d442f6f7a1413c9d7da2c870f16050cf435496"} Oct 30 21:30:08.526610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.526596 4526 scope.go:122] "RemoveContainer" containerID="7d47a8b678828107f63e7fd0a5d442f6f7a1413c9d7da2c870f16050cf435496" Oct 30 21:30:08.570315 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:08.570295 4526 scope.go:122] "RemoveContainer" containerID="76dc0663881f80c371b646287208f77cc385e586129c1462d027d6ea9bf9c770" Oct 30 21:30:09.533350 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.533322 4526 scope.go:122] "RemoveContainer" containerID="32849177e57ef7c27179169b7f3ed93e311b712ca430fb4caaef57241bd945f5" Oct 30 21:30:09.533785 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:09.533663 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj_odh-ai-gateway-infra(3b73e00e-1d35-4e34-ab57-fc9ab9e17c55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" Oct 30 21:30:09.535097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.535063 4526 generic.go:356] "Generic (PLEG): container finished" podID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerID="624ce90104322413b525bc104e2ab0a85f1c58b974425ba8fa24e824c04a8d62" exitCode=60 Oct 30 21:30:09.535205 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.535145 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" event={"ID":"f3008baf-ffa9-41d9-bd5d-465ebfdbb813","Type":"ContainerDied","Data":"624ce90104322413b525bc104e2ab0a85f1c58b974425ba8fa24e824c04a8d62"} Oct 30 21:30:09.535285 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.535203 4526 scope.go:122] "RemoveContainer" containerID="f4ae4691a6b0f55921afc6af474858f1b470a1b7c7b242121d39071cf489f381" Oct 30 21:30:09.535443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.535429 4526 scope.go:122] "RemoveContainer" containerID="624ce90104322413b525bc104e2ab0a85f1c58b974425ba8fa24e824c04a8d62" Oct 30 21:30:09.535751 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:09.535732 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc_odh-ai-gateway-infra(f3008baf-ffa9-41d9-bd5d-465ebfdbb813)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" Oct 30 21:30:09.536862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.536846 4526 scope.go:122] "RemoveContainer" containerID="3a4d2deb50ca493fe09de2eabfdd107e1da38c2d3550b8fbbec10c34e2d3d44c" Oct 30 21:30:09.537086 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:09.537061 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-c9vfn_odh-ai-gateway-infra(04683c2f-82e5-4131-b6cf-7a60a5045565)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" Oct 30 21:30:09.538435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.538414 4526 generic.go:356] "Generic (PLEG): container finished" podID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerID="f81ab444f008b28186ab146d4d33a1c1dbc9f605eae0a2665f3e8d478d96714f" exitCode=60 Oct 30 21:30:09.538570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.538517 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" event={"ID":"e0025b80-00e8-4fe1-88fa-69429daa452e","Type":"ContainerDied","Data":"f81ab444f008b28186ab146d4d33a1c1dbc9f605eae0a2665f3e8d478d96714f"} Oct 30 21:30:09.538767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.538754 4526 scope.go:122] "RemoveContainer" containerID="f81ab444f008b28186ab146d4d33a1c1dbc9f605eae0a2665f3e8d478d96714f" Oct 30 21:30:09.539012 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:09.538987 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz_odh-ai-gateway-infra(e0025b80-00e8-4fe1-88fa-69429daa452e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" Oct 30 21:30:09.540437 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.540415 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" event={"ID":"00d304b1-7215-493c-a773-26c638be6775","Type":"ContainerStarted","Data":"22006ea96e6d898b74664dca4ca157efa49dae516c2d5958184c8c241277a547"} Oct 30 21:30:09.555057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:09.555044 4526 scope.go:122] "RemoveContainer" containerID="eb807b0cbf41b4de8d89c35e273f6be94c06e76b482faa5df89a00ec1c2c0f38" Oct 30 21:30:10.547433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:10.547403 4526 scope.go:122] "RemoveContainer" containerID="624ce90104322413b525bc104e2ab0a85f1c58b974425ba8fa24e824c04a8d62" Oct 30 21:30:10.547910 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:10.547682 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc_odh-ai-gateway-infra(f3008baf-ffa9-41d9-bd5d-465ebfdbb813)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" Oct 30 21:30:10.548894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:10.548873 4526 scope.go:122] "RemoveContainer" containerID="f81ab444f008b28186ab146d4d33a1c1dbc9f605eae0a2665f3e8d478d96714f" Oct 30 21:30:10.549126 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:10.549104 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz_odh-ai-gateway-infra(e0025b80-00e8-4fe1-88fa-69429daa452e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" Oct 30 21:30:10.550398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:10.550378 4526 generic.go:356] "Generic (PLEG): container finished" podID="00d304b1-7215-493c-a773-26c638be6775" containerID="22006ea96e6d898b74664dca4ca157efa49dae516c2d5958184c8c241277a547" exitCode=60 Oct 30 21:30:10.550474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:10.550423 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" event={"ID":"00d304b1-7215-493c-a773-26c638be6775","Type":"ContainerDied","Data":"22006ea96e6d898b74664dca4ca157efa49dae516c2d5958184c8c241277a547"} Oct 30 21:30:10.550540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:10.550512 4526 scope.go:122] "RemoveContainer" containerID="7d47a8b678828107f63e7fd0a5d442f6f7a1413c9d7da2c870f16050cf435496" Oct 30 21:30:10.550685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:10.550670 4526 scope.go:122] "RemoveContainer" containerID="22006ea96e6d898b74664dca4ca157efa49dae516c2d5958184c8c241277a547" Oct 30 21:30:10.550930 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:10.550909 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz_odh-ai-gateway-infra(00d304b1-7215-493c-a773-26c638be6775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" podUID="00d304b1-7215-493c-a773-26c638be6775" Oct 30 21:30:11.557472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:11.557441 4526 scope.go:122] "RemoveContainer" containerID="22006ea96e6d898b74664dca4ca157efa49dae516c2d5958184c8c241277a547" Oct 30 21:30:11.557908 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:11.557726 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz_odh-ai-gateway-infra(00d304b1-7215-493c-a773-26c638be6775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" podUID="00d304b1-7215-493c-a773-26c638be6775" Oct 30 21:30:12.478368 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:12.478331 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:30:18.978071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:18.978038 4526 scope.go:122] "RemoveContainer" containerID="32849177e57ef7c27179169b7f3ed93e311b712ca430fb4caaef57241bd945f5" Oct 30 21:30:19.610856 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:19.610769 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" event={"ID":"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55","Type":"ContainerStarted","Data":"f565b5855e8b3a01b6f45430c71bada3211fb315c8ed1b537b04b901d54597cf"} Oct 30 21:30:19.727166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:19.727070 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" podStartSLOduration=14.421165443 podStartE2EDuration="19.727051058s" totalImagesPullingTime="5.305885615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:19.723102891 +0000 UTC m=+809.856476088" watchObservedRunningTime="2026-10-30 21:30:19.727051058 +0000 UTC m=+809.860424270" Oct 30 21:30:20.067648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.066712 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:48522->10.133.0.36:15021: read: connection reset by peer" start-of-body= Oct 30 21:30:20.067648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.066794 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": read tcp 10.133.0.2:48522->10.133.0.36:15021: read: connection reset by peer" Oct 30 21:30:20.617882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.617845 4526 generic.go:356] "Generic (PLEG): container finished" podID="46444220-52f6-4122-b602-6a2a6531a4de" containerID="4aecfe60cd248537425ce239a0062ca1a0c1ae43814424e8b120108aca8274c4" exitCode=0 Oct 30 21:30:20.618062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.617914 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" event={"ID":"46444220-52f6-4122-b602-6a2a6531a4de","Type":"ContainerDied","Data":"4aecfe60cd248537425ce239a0062ca1a0c1ae43814424e8b120108aca8274c4"} Oct 30 21:30:20.618413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.618395 4526 scope.go:122] "RemoveContainer" containerID="4aecfe60cd248537425ce239a0062ca1a0c1ae43814424e8b120108aca8274c4" Oct 30 21:30:20.619970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.619943 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:30:20.620067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.620020 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:30:20.620067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.620061 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:30:20.620156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.620079 4526 generic.go:356] "Generic (PLEG): container finished" podID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerID="f565b5855e8b3a01b6f45430c71bada3211fb315c8ed1b537b04b901d54597cf" exitCode=60 Oct 30 21:30:20.620156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.620105 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" event={"ID":"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55","Type":"ContainerDied","Data":"f565b5855e8b3a01b6f45430c71bada3211fb315c8ed1b537b04b901d54597cf"} Oct 30 21:30:20.620156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.620139 4526 scope.go:122] "RemoveContainer" containerID="32849177e57ef7c27179169b7f3ed93e311b712ca430fb4caaef57241bd945f5" Oct 30 21:30:20.620512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.620467 4526 scope.go:122] "RemoveContainer" containerID="f565b5855e8b3a01b6f45430c71bada3211fb315c8ed1b537b04b901d54597cf" Oct 30 21:30:20.620763 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:20.620747 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj_odh-ai-gateway-infra(3b73e00e-1d35-4e34-ab57-fc9ab9e17c55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" Oct 30 21:30:20.734774 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:20.734743 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj"] Oct 30 21:30:21.634952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.634909 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" event={"ID":"46444220-52f6-4122-b602-6a2a6531a4de","Type":"ContainerStarted","Data":"4e852ba86c135a80be8188e11d50857a5900e2c930a3f452edfe844a9f9c5408"} Oct 30 21:30:21.861910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.861879 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:21.947706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.947596 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-maas-api-service-ca\") pod \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\" (UID: \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\") " Oct 30 21:30:21.947897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.947745 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-kube-api-access-rdxqt\" (UniqueName: \"kubernetes.io/projected/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-kube-api-access-rdxqt\") pod \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\" (UID: \"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55\") " Oct 30 21:30:21.947995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.947951 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-maas-api-service-ca" pod "3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" (UID: "3b73e00e-1d35-4e34-ab57-fc9ab9e17c55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:21.948113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.948098 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:21.950165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.950142 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-kube-api-access-rdxqt" pod "3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" (UID: "3b73e00e-1d35-4e34-ab57-fc9ab9e17c55"). InnerVolumeSpecName "kube-api-access-rdxqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:21.978951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.978913 4526 scope.go:122] "RemoveContainer" containerID="22006ea96e6d898b74664dca4ca157efa49dae516c2d5958184c8c241277a547" Oct 30 21:30:21.979139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:21.978958 4526 scope.go:122] "RemoveContainer" containerID="624ce90104322413b525bc104e2ab0a85f1c58b974425ba8fa24e824c04a8d62" Oct 30 21:30:22.049235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.049200 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdxqt\" (UniqueName: \"kubernetes.io/projected/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55-kube-api-access-rdxqt\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:22.143657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.143622 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:30:22.143953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.143930 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:30:22.144001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.143982 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 21:30:22.644970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.644930 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" event={"ID":"00d304b1-7215-493c-a773-26c638be6775","Type":"ContainerStarted","Data":"3abf2d17baa4be46ba6f7b1799de4479e6d69a5b316e9398584e887f8f298362"} Oct 30 21:30:22.646411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.646390 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" Oct 30 21:30:22.646411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.646399 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj" event={"ID":"3b73e00e-1d35-4e34-ab57-fc9ab9e17c55","Type":"ContainerDied","Data":"f43cd00af5089a0ac93f033a1b68aff9fcbba3cd3f1f0599781b930c7d085751"} Oct 30 21:30:22.646634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.646457 4526 scope.go:122] "RemoveContainer" containerID="f565b5855e8b3a01b6f45430c71bada3211fb315c8ed1b537b04b901d54597cf" Oct 30 21:30:22.648568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.648537 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" event={"ID":"f3008baf-ffa9-41d9-bd5d-465ebfdbb813","Type":"ContainerStarted","Data":"b54ed1e41ec899c371db1679df5a5c807a0a365e380f40dfe4c85f965f34d2d3"} Oct 30 21:30:22.891765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.891729 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj"] Oct 30 21:30:22.908562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.896216 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889930-lmdwj"] Oct 30 21:30:22.978259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:22.978233 4526 scope.go:122] "RemoveContainer" containerID="3a4d2deb50ca493fe09de2eabfdd107e1da38c2d3550b8fbbec10c34e2d3d44c" Oct 30 21:30:23.144017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.143991 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:30:23.144149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.144049 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 21:30:23.656738 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.656708 4526 generic.go:356] "Generic (PLEG): container finished" podID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerID="b54ed1e41ec899c371db1679df5a5c807a0a365e380f40dfe4c85f965f34d2d3" exitCode=60 Oct 30 21:30:23.657249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.656787 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" event={"ID":"f3008baf-ffa9-41d9-bd5d-465ebfdbb813","Type":"ContainerDied","Data":"b54ed1e41ec899c371db1679df5a5c807a0a365e380f40dfe4c85f965f34d2d3"} Oct 30 21:30:23.657249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.656849 4526 scope.go:122] "RemoveContainer" containerID="624ce90104322413b525bc104e2ab0a85f1c58b974425ba8fa24e824c04a8d62" Oct 30 21:30:23.657249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.657220 4526 scope.go:122] "RemoveContainer" containerID="b54ed1e41ec899c371db1679df5a5c807a0a365e380f40dfe4c85f965f34d2d3" Oct 30 21:30:23.657566 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:23.657546 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc_odh-ai-gateway-infra(f3008baf-ffa9-41d9-bd5d-465ebfdbb813)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" Oct 30 21:30:23.658775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.658754 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" event={"ID":"04683c2f-82e5-4131-b6cf-7a60a5045565","Type":"ContainerStarted","Data":"2d7df2082433640191a530e980bcb227c95aeec7697d4dd8fa29a783c8faa753"} Oct 30 21:30:23.660405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.660382 4526 generic.go:356] "Generic (PLEG): container finished" podID="00d304b1-7215-493c-a773-26c638be6775" containerID="3abf2d17baa4be46ba6f7b1799de4479e6d69a5b316e9398584e887f8f298362" exitCode=60 Oct 30 21:30:23.660504 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.660451 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" event={"ID":"00d304b1-7215-493c-a773-26c638be6775","Type":"ContainerDied","Data":"3abf2d17baa4be46ba6f7b1799de4479e6d69a5b316e9398584e887f8f298362"} Oct 30 21:30:23.660814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.660759 4526 scope.go:122] "RemoveContainer" containerID="3abf2d17baa4be46ba6f7b1799de4479e6d69a5b316e9398584e887f8f298362" Oct 30 21:30:23.661007 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:23.660990 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz_odh-ai-gateway-infra(00d304b1-7215-493c-a773-26c638be6775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" podUID="00d304b1-7215-493c-a773-26c638be6775" Oct 30 21:30:23.668870 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.668852 4526 scope.go:122] "RemoveContainer" containerID="22006ea96e6d898b74664dca4ca157efa49dae516c2d5958184c8c241277a547" Oct 30 21:30:23.793035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.793004 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz"] Oct 30 21:30:23.978206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.978124 4526 scope.go:122] "RemoveContainer" containerID="f81ab444f008b28186ab146d4d33a1c1dbc9f605eae0a2665f3e8d478d96714f" Oct 30 21:30:23.985738 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:23.985708 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" path="/var/lib/kubelet/pods/3b73e00e-1d35-4e34-ab57-fc9ab9e17c55/volumes" Oct 30 21:30:24.013004 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.009632 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" podStartSLOduration=18.585798556 podStartE2EDuration="24.009612298s" totalImagesPullingTime="5.423813742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:24.007909533 +0000 UTC m=+814.141282731" watchObservedRunningTime="2026-10-30 21:30:24.009612298 +0000 UTC m=+814.142985496" Oct 30 21:30:24.026464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.026433 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc"] Oct 30 21:30:24.143623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.143586 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:30:24.143816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.143641 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 21:30:24.667625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.667587 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" event={"ID":"e0025b80-00e8-4fe1-88fa-69429daa452e","Type":"ContainerStarted","Data":"2db438a98f495df68168cc409bf1c2e06b3ad6d3a8ef85cc4203ea2f3f0ba43e"} Oct 30 21:30:24.673151 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.673126 4526 generic.go:356] "Generic (PLEG): container finished" podID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerID="2d7df2082433640191a530e980bcb227c95aeec7697d4dd8fa29a783c8faa753" exitCode=60 Oct 30 21:30:24.673258 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.673173 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" event={"ID":"04683c2f-82e5-4131-b6cf-7a60a5045565","Type":"ContainerDied","Data":"2d7df2082433640191a530e980bcb227c95aeec7697d4dd8fa29a783c8faa753"} Oct 30 21:30:24.673258 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.673213 4526 scope.go:122] "RemoveContainer" containerID="3a4d2deb50ca493fe09de2eabfdd107e1da38c2d3550b8fbbec10c34e2d3d44c" Oct 30 21:30:24.673554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.673539 4526 scope.go:122] "RemoveContainer" containerID="2d7df2082433640191a530e980bcb227c95aeec7697d4dd8fa29a783c8faa753" Oct 30 21:30:24.673813 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:24.673795 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-c9vfn_odh-ai-gateway-infra(04683c2f-82e5-4131-b6cf-7a60a5045565)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" Oct 30 21:30:24.867991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.867967 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:24.887658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.887636 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:24.979914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.979839 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-maas-api-service-ca\") pod \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\" (UID: \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\") " Oct 30 21:30:24.979914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.979887 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d304b1-7215-493c-a773-26c638be6775-kube-api-access-z9qzw\" (UniqueName: \"kubernetes.io/projected/00d304b1-7215-493c-a773-26c638be6775-kube-api-access-z9qzw\") pod \"00d304b1-7215-493c-a773-26c638be6775\" (UID: \"00d304b1-7215-493c-a773-26c638be6775\") " Oct 30 21:30:24.979914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.979911 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d304b1-7215-493c-a773-26c638be6775-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d304b1-7215-493c-a773-26c638be6775-maas-api-service-ca\") pod \"00d304b1-7215-493c-a773-26c638be6775\" (UID: \"00d304b1-7215-493c-a773-26c638be6775\") " Oct 30 21:30:24.980190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.980050 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-kube-api-access-drw5n\" (UniqueName: \"kubernetes.io/projected/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-kube-api-access-drw5n\") pod \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\" (UID: \"f3008baf-ffa9-41d9-bd5d-465ebfdbb813\") " Oct 30 21:30:24.980253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.980185 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-maas-api-service-ca" pod "f3008baf-ffa9-41d9-bd5d-465ebfdbb813" (UID: "f3008baf-ffa9-41d9-bd5d-465ebfdbb813"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.980306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.980289 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d304b1-7215-493c-a773-26c638be6775-maas-api-service-ca" pod "00d304b1-7215-493c-a773-26c638be6775" (UID: "00d304b1-7215-493c-a773-26c638be6775"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.980420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.980405 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.980458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.980422 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d304b1-7215-493c-a773-26c638be6775-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:24.982202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.982182 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d304b1-7215-493c-a773-26c638be6775-kube-api-access-z9qzw" pod "00d304b1-7215-493c-a773-26c638be6775" (UID: "00d304b1-7215-493c-a773-26c638be6775"). InnerVolumeSpecName "kube-api-access-z9qzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:24.982274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:24.982218 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-kube-api-access-drw5n" pod "f3008baf-ffa9-41d9-bd5d-465ebfdbb813" (UID: "f3008baf-ffa9-41d9-bd5d-465ebfdbb813"). InnerVolumeSpecName "kube-api-access-drw5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:25.023563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.023455 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn"] Oct 30 21:30:25.081866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.081841 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drw5n\" (UniqueName: \"kubernetes.io/projected/f3008baf-ffa9-41d9-bd5d-465ebfdbb813-kube-api-access-drw5n\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:25.081866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.081867 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9qzw\" (UniqueName: \"kubernetes.io/projected/00d304b1-7215-493c-a773-26c638be6775-kube-api-access-z9qzw\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:25.143528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.143475 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:30:25.143683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.143545 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 21:30:25.681210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.681174 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" Oct 30 21:30:25.681701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.681179 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz" event={"ID":"00d304b1-7215-493c-a773-26c638be6775","Type":"ContainerDied","Data":"5b874a087cbe84b67bf44128374cbba5043359ca9cf0efb622ad2f794f3c427b"} Oct 30 21:30:25.681701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.681388 4526 scope.go:122] "RemoveContainer" containerID="3abf2d17baa4be46ba6f7b1799de4479e6d69a5b316e9398584e887f8f298362" Oct 30 21:30:25.682918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.682701 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" event={"ID":"f3008baf-ffa9-41d9-bd5d-465ebfdbb813","Type":"ContainerDied","Data":"0ac5a39358355b55c55dc570babc16a9f073fa6cc305a00b4e2159ffbd6f2d85"} Oct 30 21:30:25.682918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.682720 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc" Oct 30 21:30:25.688709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.688682 4526 generic.go:356] "Generic (PLEG): container finished" podID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerID="2db438a98f495df68168cc409bf1c2e06b3ad6d3a8ef85cc4203ea2f3f0ba43e" exitCode=60 Oct 30 21:30:25.688830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.688735 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" event={"ID":"e0025b80-00e8-4fe1-88fa-69429daa452e","Type":"ContainerDied","Data":"2db438a98f495df68168cc409bf1c2e06b3ad6d3a8ef85cc4203ea2f3f0ba43e"} Oct 30 21:30:25.689061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.689046 4526 scope.go:122] "RemoveContainer" containerID="2db438a98f495df68168cc409bf1c2e06b3ad6d3a8ef85cc4203ea2f3f0ba43e" Oct 30 21:30:25.689316 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:25.689279 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz_odh-ai-gateway-infra(e0025b80-00e8-4fe1-88fa-69429daa452e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" Oct 30 21:30:25.704511 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.699869 4526 scope.go:122] "RemoveContainer" containerID="b54ed1e41ec899c371db1679df5a5c807a0a365e380f40dfe4c85f965f34d2d3" Oct 30 21:30:25.724225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.724097 4526 scope.go:122] "RemoveContainer" containerID="f81ab444f008b28186ab146d4d33a1c1dbc9f605eae0a2665f3e8d478d96714f" Oct 30 21:30:25.792727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.792700 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz"] Oct 30 21:30:25.844074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.844045 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc"] Oct 30 21:30:25.846107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.846084 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889930-zlwdc"] Oct 30 21:30:25.882659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.882641 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:25.985292 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.985221 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" path="/var/lib/kubelet/pods/f3008baf-ffa9-41d9-bd5d-465ebfdbb813/volumes" Oct 30 21:30:25.990567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.990545 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04683c2f-82e5-4131-b6cf-7a60a5045565-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04683c2f-82e5-4131-b6cf-7a60a5045565-maas-api-service-ca\") pod \"04683c2f-82e5-4131-b6cf-7a60a5045565\" (UID: \"04683c2f-82e5-4131-b6cf-7a60a5045565\") " Oct 30 21:30:25.990680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.990600 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04683c2f-82e5-4131-b6cf-7a60a5045565-kube-api-access-tqk24\" (UniqueName: \"kubernetes.io/projected/04683c2f-82e5-4131-b6cf-7a60a5045565-kube-api-access-tqk24\") pod \"04683c2f-82e5-4131-b6cf-7a60a5045565\" (UID: \"04683c2f-82e5-4131-b6cf-7a60a5045565\") " Oct 30 21:30:25.990918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.990891 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04683c2f-82e5-4131-b6cf-7a60a5045565-maas-api-service-ca" pod "04683c2f-82e5-4131-b6cf-7a60a5045565" (UID: "04683c2f-82e5-4131-b6cf-7a60a5045565"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:25.992755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:25.992734 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04683c2f-82e5-4131-b6cf-7a60a5045565-kube-api-access-tqk24" pod "04683c2f-82e5-4131-b6cf-7a60a5045565" (UID: "04683c2f-82e5-4131-b6cf-7a60a5045565"). InnerVolumeSpecName "kube-api-access-tqk24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:26.065839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.065806 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz"] Oct 30 21:30:26.069259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.069234 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889930-mwtxz"] Oct 30 21:30:26.092163 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.092137 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04683c2f-82e5-4131-b6cf-7a60a5045565-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.092163 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.092158 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqk24\" (UniqueName: \"kubernetes.io/projected/04683c2f-82e5-4131-b6cf-7a60a5045565-kube-api-access-tqk24\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:26.144439 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.144411 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:30:26.144607 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.144453 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 21:30:26.696049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.696026 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" Oct 30 21:30:26.696581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.696057 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn" event={"ID":"04683c2f-82e5-4131-b6cf-7a60a5045565","Type":"ContainerDied","Data":"58889ffaa029e767ff70167c0f6422ddcda6a9e4ea20c2cecd68748b46bd5d6c"} Oct 30 21:30:26.696581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.696106 4526 scope.go:122] "RemoveContainer" containerID="2d7df2082433640191a530e980bcb227c95aeec7697d4dd8fa29a783c8faa753" Oct 30 21:30:26.859428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.859402 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn"] Oct 30 21:30:26.863462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.863434 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-c9vfn"] Oct 30 21:30:26.867196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:26.867177 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:27.015527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.015426 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0025b80-00e8-4fe1-88fa-69429daa452e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0025b80-00e8-4fe1-88fa-69429daa452e-maas-api-service-ca\") pod \"e0025b80-00e8-4fe1-88fa-69429daa452e\" (UID: \"e0025b80-00e8-4fe1-88fa-69429daa452e\") " Oct 30 21:30:27.015527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.015473 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0025b80-00e8-4fe1-88fa-69429daa452e-kube-api-access-4hqtn\" (UniqueName: \"kubernetes.io/projected/e0025b80-00e8-4fe1-88fa-69429daa452e-kube-api-access-4hqtn\") pod \"e0025b80-00e8-4fe1-88fa-69429daa452e\" (UID: \"e0025b80-00e8-4fe1-88fa-69429daa452e\") " Oct 30 21:30:27.015836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.015812 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0025b80-00e8-4fe1-88fa-69429daa452e-maas-api-service-ca" pod "e0025b80-00e8-4fe1-88fa-69429daa452e" (UID: "e0025b80-00e8-4fe1-88fa-69429daa452e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:27.017753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.017731 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0025b80-00e8-4fe1-88fa-69429daa452e-kube-api-access-4hqtn" pod "e0025b80-00e8-4fe1-88fa-69429daa452e" (UID: "e0025b80-00e8-4fe1-88fa-69429daa452e"). InnerVolumeSpecName "kube-api-access-4hqtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:27.116921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.116879 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0025b80-00e8-4fe1-88fa-69429daa452e-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:27.116921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.116905 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hqtn\" (UniqueName: \"kubernetes.io/projected/e0025b80-00e8-4fe1-88fa-69429daa452e-kube-api-access-4hqtn\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:30:27.147628 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.147601 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:30:27.148014 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.147992 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:30:27.148599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.148578 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:30:27.709155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.709128 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" Oct 30 21:30:27.709647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.709160 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz" event={"ID":"e0025b80-00e8-4fe1-88fa-69429daa452e","Type":"ContainerDied","Data":"31efad9b9617e72aba2f0a061f8a9716c17cb62a10d976eba36129436cf67ebc"} Oct 30 21:30:27.709647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.709217 4526 scope.go:122] "RemoveContainer" containerID="2db438a98f495df68168cc409bf1c2e06b3ad6d3a8ef85cc4203ea2f3f0ba43e" Oct 30 21:30:27.871771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.871736 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz"] Oct 30 21:30:27.875651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.875628 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889930-h4tnz"] Oct 30 21:30:27.984942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.984863 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d304b1-7215-493c-a773-26c638be6775" path="/var/lib/kubelet/pods/00d304b1-7215-493c-a773-26c638be6775/volumes" Oct 30 21:30:27.985309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.985296 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" path="/var/lib/kubelet/pods/04683c2f-82e5-4131-b6cf-7a60a5045565/volumes" Oct 30 21:30:27.985691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:27.985677 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" path="/var/lib/kubelet/pods/e0025b80-00e8-4fe1-88fa-69429daa452e/volumes" Oct 30 21:30:33.880412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880383 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj"] Oct 30 21:30:33.880895 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880881 4526 cpu_manager.go:414] "containerMap: removing container" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:33.880938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880897 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:33.880938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880908 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:33.880938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880914 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:33.880938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880923 4526 cpu_manager.go:414] "containerMap: removing container" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:33.880938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880928 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:33.880938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880936 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880940 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880951 4526 cpu_manager.go:414] "containerMap: removing container" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880955 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880965 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880970 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880980 4526 cpu_manager.go:414] "containerMap: removing container" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880985 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880993 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.880998 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881006 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881010 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881020 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881024 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881030 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881034 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881109 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881116 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881123 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881128 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b73e00e-1d35-4e34-ab57-fc9ab9e17c55" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881134 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881141 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881145 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881150 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881156 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881160 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881165 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:33.881604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.881170 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:33.885905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.885888 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:33.960397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.960364 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj"] Oct 30 21:30:33.988178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.988141 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"kube-root-ca.crt\"" Oct 30 21:30:33.988346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.988302 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"default-dockercfg-hqthv\"" Oct 30 21:30:33.988346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.988306 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"openshift-service-ca.crt\"" Oct 30 21:30:33.988436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.988358 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"simulator-w2-446ca2-kserve-self-signed-certs\"" Oct 30 21:30:33.989717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.989699 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-dshm\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:33.989760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.989741 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-home\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:33.989894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.989878 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-model-cache\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:33.989934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.989923 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tls-certs\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:33.989967 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.989947 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tmp-dir\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:33.990045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:33.990032 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ddn5\" (UniqueName: \"kubernetes.io/projected/32765e94-ffd5-4160-b1bd-0636b0f5ac86-kube-api-access-6ddn5\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.091532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.091472 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-dshm\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.091720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.091564 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-home\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.091720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.091623 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-model-cache\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.091720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.091640 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tls-certs\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.091720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.091659 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tmp-dir\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.091943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.091771 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddn5\" (UniqueName: \"kubernetes.io/projected/32765e94-ffd5-4160-b1bd-0636b0f5ac86-kube-api-access-6ddn5\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.092049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.092026 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-model-cache\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.092132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.092049 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-home\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.092356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.092335 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tmp-dir\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.093991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.093968 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-dshm\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.094426 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.094407 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tls-certs\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.150681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.150600 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddn5\" (UniqueName: \"kubernetes.io/projected/32765e94-ffd5-4160-b1bd-0636b0f5ac86-kube-api-access-6ddn5\") pod \"simulator-w2-446ca2-kserve-7467554bb9-h4rfj\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.197646 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.197605 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:34.576432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.576407 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj"] Oct 30 21:30:34.578937 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:30:34.578907 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32765e94_ffd5_4160_b1bd_0636b0f5ac86.slice/crio-99b6e78b33657fcd7ed82c2ef7b132025968e718073bbf3e97436f3142fdd7a9 WatchSource:0}: Error finding container 99b6e78b33657fcd7ed82c2ef7b132025968e718073bbf3e97436f3142fdd7a9: Status 404 returned error can't find the container with id 99b6e78b33657fcd7ed82c2ef7b132025968e718073bbf3e97436f3142fdd7a9 Oct 30 21:30:34.749943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.749908 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" event={"ID":"32765e94-ffd5-4160-b1bd-0636b0f5ac86","Type":"ContainerStarted","Data":"a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651"} Oct 30 21:30:34.750089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.749954 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" event={"ID":"32765e94-ffd5-4160-b1bd-0636b0f5ac86","Type":"ContainerStarted","Data":"99b6e78b33657fcd7ed82c2ef7b132025968e718073bbf3e97436f3142fdd7a9"} Oct 30 21:30:34.865587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:34.865468 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podStartSLOduration=1.865452106 podStartE2EDuration="1.865452106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:34.863916451 +0000 UTC m=+824.997289649" watchObservedRunningTime="2026-10-30 21:30:34.865452106 +0000 UTC m=+824.998825303" Oct 30 21:30:35.198152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:35.198070 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:42.477468 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:30:42.477431 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:30:44.198476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:44.198447 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:44.211594 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:44.211573 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:44.861441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:44.861411 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:30:56.177243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177203 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp"] Oct 30 21:30:56.177918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177896 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:56.178003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177919 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0025b80-00e8-4fe1-88fa-69429daa452e" containerName="cleanup" Oct 30 21:30:56.178003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177938 4526 cpu_manager.go:414] "containerMap: removing container" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:56.178003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177945 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:56.178003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177960 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:56.178003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177969 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:56.178003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177989 4526 cpu_manager.go:414] "containerMap: removing container" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:56.178003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.177996 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:56.178330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.178119 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3008baf-ffa9-41d9-bd5d-465ebfdbb813" containerName="cleanup" Oct 30 21:30:56.178330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.178138 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d304b1-7215-493c-a773-26c638be6775" containerName="cleanup" Oct 30 21:30:56.178330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.178148 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="04683c2f-82e5-4131-b6cf-7a60a5045565" containerName="cleanup" Oct 30 21:30:56.182784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.182764 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.248898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.248856 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ldh\" (UniqueName: \"kubernetes.io/projected/8d354f8a-4ff3-466c-875f-d7cda66177e7-kube-api-access-f9ldh\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.249072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.249010 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-home\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.249072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.249038 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-model-cache\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.249072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.249065 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-dshm\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.249243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.249134 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d354f8a-4ff3-466c-875f-d7cda66177e7-tls-certs\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.249305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.249249 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-tmp-dir\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.284755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.284727 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e4dadd\"/\"openshift-service-ca.crt\"" Oct 30 21:30:56.284755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.284744 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e4dadd\"/\"premium-w1-e4dadd-kserve-self-signed-certs\"" Oct 30 21:30:56.284755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.284727 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e4dadd\"/\"kube-root-ca.crt\"" Oct 30 21:30:56.284973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.284833 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e4dadd\"/\"default-dockercfg-45994\"" Oct 30 21:30:56.288632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.288605 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp"] Oct 30 21:30:56.350029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.349988 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ldh\" (UniqueName: \"kubernetes.io/projected/8d354f8a-4ff3-466c-875f-d7cda66177e7-kube-api-access-f9ldh\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.350231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.350071 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-home\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.350231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.350096 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-model-cache\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.350231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.350114 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-dshm\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.350231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.350134 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d354f8a-4ff3-466c-875f-d7cda66177e7-tls-certs\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.350437 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.350324 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-tmp-dir\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.350598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.350578 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-model-cache\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.350661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.350607 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-home\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.350716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.350691 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-tmp-dir\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.352735 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.352711 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-dshm\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.353190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.353174 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d354f8a-4ff3-466c-875f-d7cda66177e7-tls-certs\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.406822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.406784 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ldh\" (UniqueName: \"kubernetes.io/projected/8d354f8a-4ff3-466c-875f-d7cda66177e7-kube-api-access-f9ldh\") pod \"premium-w1-e4dadd-kserve-66cc9dddb7-8thdp\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.541281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.541246 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:30:56.931656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:56.931630 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp"] Oct 30 21:30:57.915477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:57.915439 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" event={"ID":"8d354f8a-4ff3-466c-875f-d7cda66177e7","Type":"ContainerStarted","Data":"a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d"} Oct 30 21:30:57.915477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:57.915509 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" event={"ID":"8d354f8a-4ff3-466c-875f-d7cda66177e7","Type":"ContainerStarted","Data":"08258368eb07f8e2b1206ce87b5770a999dfbc7f92deb63632fc63a40c29ea97"} Oct 30 21:30:58.037577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:58.037516 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podStartSLOduration=2.037472064 podStartE2EDuration="2.037472064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:58.035533536 +0000 UTC m=+848.168906731" watchObservedRunningTime="2026-10-30 21:30:58.037472064 +0000 UTC m=+848.170845261" Oct 30 21:30:58.541945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:30:58.541905 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:31:06.542471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:06.542389 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:31:06.554915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:06.554894 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:31:06.980865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:06.980790 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:31:12.479166 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:31:12.479131 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:31:25.252506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.252459 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw"] Oct 30 21:31:25.256397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.256379 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.361082 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.361048 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"kube-root-ca.crt\"" Oct 30 21:31:25.361082 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.361077 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"openshift-service-ca.crt\"" Oct 30 21:31:25.361338 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.361083 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"simulator-w6-89c0fd-kserve-self-signed-certs\"" Oct 30 21:31:25.361338 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.361232 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"default-dockercfg-mmr5s\"" Oct 30 21:31:25.363810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.363787 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw"] Oct 30 21:31:25.372920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.372894 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-tmp-dir\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.373072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.373018 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-home\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.373072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.373038 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-model-cache\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.373144 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.373103 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklxn\" (UniqueName: \"kubernetes.io/projected/12bbf005-fe98-432f-a6fc-13e370af59df-kube-api-access-qklxn\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.373178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.373149 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-dshm\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.373178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.373172 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12bbf005-fe98-432f-a6fc-13e370af59df-tls-certs\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474225 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklxn\" (UniqueName: \"kubernetes.io/projected/12bbf005-fe98-432f-a6fc-13e370af59df-kube-api-access-qklxn\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474278 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-dshm\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474302 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12bbf005-fe98-432f-a6fc-13e370af59df-tls-certs\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474353 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-tmp-dir\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474428 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-home\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474443 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-model-cache\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474811 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-model-cache\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474852 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-tmp-dir\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.474976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.474911 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-home\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.476821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.476803 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-dshm\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.477233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.477214 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12bbf005-fe98-432f-a6fc-13e370af59df-tls-certs\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.531049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.530969 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklxn\" (UniqueName: \"kubernetes.io/projected/12bbf005-fe98-432f-a6fc-13e370af59df-kube-api-access-qklxn\") pod \"simulator-w6-89c0fd-kserve-58546d97ff-w4kjw\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.568885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.568851 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:25.957334 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:25.957311 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw"] Oct 30 21:31:25.960054 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:31:25.960026 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12bbf005_fe98_432f_a6fc_13e370af59df.slice/crio-b407d39ccdf6061eb86d052d74242f6433c65a14044c594c87a98afd3d26f5a0 WatchSource:0}: Error finding container b407d39ccdf6061eb86d052d74242f6433c65a14044c594c87a98afd3d26f5a0: Status 404 returned error can't find the container with id b407d39ccdf6061eb86d052d74242f6433c65a14044c594c87a98afd3d26f5a0 Oct 30 21:31:26.072307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:26.072273 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" event={"ID":"12bbf005-fe98-432f-a6fc-13e370af59df","Type":"ContainerStarted","Data":"5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8"} Oct 30 21:31:26.072513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:26.072315 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" event={"ID":"12bbf005-fe98-432f-a6fc-13e370af59df","Type":"ContainerStarted","Data":"b407d39ccdf6061eb86d052d74242f6433c65a14044c594c87a98afd3d26f5a0"} Oct 30 21:31:26.187618 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:26.187564 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podStartSLOduration=1.187550415 podStartE2EDuration="1.187550415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:26.184849696 +0000 UTC m=+876.318222905" watchObservedRunningTime="2026-10-30 21:31:26.187550415 +0000 UTC m=+876.320923677" Oct 30 21:31:26.569310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:26.569271 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:35.569480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:35.569440 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:35.581653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:35.581629 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:36.137740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:31:36.137713 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:31:42.477530 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:31:42.477478 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:32:12.477373 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:32:12.477328 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:32:16.173258 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.173226 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf"] Oct 30 21:32:16.177024 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.177008 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.230020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.229994 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:32:16.230020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.229992 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:32:16.230020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.230018 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:32:16.279431 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.279396 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:32:16.281981 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.281955 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf"] Oct 30 21:32:16.329551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.329517 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqpr\" (UniqueName: \"kubernetes.io/projected/fb8864de-322e-4f94-84a1-9857a8ae5946-kube-api-access-2cqpr\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.329728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.329574 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.329728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.329710 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.371268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.371237 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n"] Oct 30 21:32:16.375043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.375023 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.430926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.430837 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.431323 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.431294 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqpr\" (UniqueName: \"kubernetes.io/projected/fb8864de-322e-4f94-84a1-9857a8ae5946-kube-api-access-2cqpr\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.431764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.431737 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.431882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.431837 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.432597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.432576 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n"] Oct 30 21:32:16.434670 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.434647 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.487943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.487909 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqpr\" (UniqueName: \"kubernetes.io/projected/fb8864de-322e-4f94-84a1-9857a8ae5946-kube-api-access-2cqpr\") pod \"maas-api-revoke-sub-c10d2545-6p5bf\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.489675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.489654 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:16.532472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.532423 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.532661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.532641 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.532709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.532673 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvq8f\" (UniqueName: \"kubernetes.io/projected/347013c5-51f5-46de-8b5e-3caf3c2848e6-kube-api-access-rvq8f\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.633853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.633818 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.634035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.633932 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.634035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.633962 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvq8f\" (UniqueName: \"kubernetes.io/projected/347013c5-51f5-46de-8b5e-3caf3c2848e6-kube-api-access-rvq8f\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.634649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.634629 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.636405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.636382 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.690044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.689975 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvq8f\" (UniqueName: \"kubernetes.io/projected/347013c5-51f5-46de-8b5e-3caf3c2848e6-kube-api-access-rvq8f\") pod \"maas-api-revoke-sub-94a8b2fe-bdx9n\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:16.861867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.861842 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf"] Oct 30 21:32:16.863980 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:32:16.863953 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8864de_322e_4f94_84a1_9857a8ae5946.slice/crio-632091f944dc01db0e14fccb797c69b3b603f38c1ff272fb5db81d1271275962 WatchSource:0}: Error finding container 632091f944dc01db0e14fccb797c69b3b603f38c1ff272fb5db81d1271275962: Status 404 returned error can't find the container with id 632091f944dc01db0e14fccb797c69b3b603f38c1ff272fb5db81d1271275962 Oct 30 21:32:16.987276 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:16.987203 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:17.359780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:17.359734 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" event={"ID":"fb8864de-322e-4f94-84a1-9857a8ae5946","Type":"ContainerStarted","Data":"82cabc52db0099f3514d2e0ab5095a2f9f4d3bdb8298c5772c639a97bdfca86c"} Oct 30 21:32:17.360225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:17.359796 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" event={"ID":"fb8864de-322e-4f94-84a1-9857a8ae5946","Type":"ContainerStarted","Data":"632091f944dc01db0e14fccb797c69b3b603f38c1ff272fb5db81d1271275962"} Oct 30 21:32:17.364715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:17.364692 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n"] Oct 30 21:32:17.366421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:32:17.366391 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347013c5_51f5_46de_8b5e_3caf3c2848e6.slice/crio-3fc151afc1e43d23a0932f384a37a5b8f9db8e042722629ccde5ccd395c0b982 WatchSource:0}: Error finding container 3fc151afc1e43d23a0932f384a37a5b8f9db8e042722629ccde5ccd395c0b982: Status 404 returned error can't find the container with id 3fc151afc1e43d23a0932f384a37a5b8f9db8e042722629ccde5ccd395c0b982 Oct 30 21:32:17.474141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:17.474090 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" podStartSLOduration=1.474075159 podStartE2EDuration="1.474075159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:17.47261238 +0000 UTC m=+927.605985578" watchObservedRunningTime="2026-10-30 21:32:17.474075159 +0000 UTC m=+927.607448357" Oct 30 21:32:18.369682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:18.369648 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" event={"ID":"347013c5-51f5-46de-8b5e-3caf3c2848e6","Type":"ContainerStarted","Data":"112d9fc325fac8bb18c16b7fb6f2000f6369678337c086bbda8354a7d452b9a8"} Oct 30 21:32:18.370118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:18.369713 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" event={"ID":"347013c5-51f5-46de-8b5e-3caf3c2848e6","Type":"ContainerStarted","Data":"3fc151afc1e43d23a0932f384a37a5b8f9db8e042722629ccde5ccd395c0b982"} Oct 30 21:32:18.371379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:18.371351 4526 generic.go:356] "Generic (PLEG): container finished" podID="fb8864de-322e-4f94-84a1-9857a8ae5946" containerID="82cabc52db0099f3514d2e0ab5095a2f9f4d3bdb8298c5772c639a97bdfca86c" exitCode=0 Oct 30 21:32:18.371520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:18.371405 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" event={"ID":"fb8864de-322e-4f94-84a1-9857a8ae5946","Type":"ContainerDied","Data":"82cabc52db0099f3514d2e0ab5095a2f9f4d3bdb8298c5772c639a97bdfca86c"} Oct 30 21:32:18.488406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:18.488348 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" podStartSLOduration=2.488334208 podStartE2EDuration="2.488334208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:18.485927465 +0000 UTC m=+928.619300666" watchObservedRunningTime="2026-10-30 21:32:18.488334208 +0000 UTC m=+928.621707406" Oct 30 21:32:19.377959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.377925 4526 generic.go:356] "Generic (PLEG): container finished" podID="347013c5-51f5-46de-8b5e-3caf3c2848e6" containerID="112d9fc325fac8bb18c16b7fb6f2000f6369678337c086bbda8354a7d452b9a8" exitCode=0 Oct 30 21:32:19.378388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.378016 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" event={"ID":"347013c5-51f5-46de-8b5e-3caf3c2848e6","Type":"ContainerDied","Data":"112d9fc325fac8bb18c16b7fb6f2000f6369678337c086bbda8354a7d452b9a8"} Oct 30 21:32:19.528255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.528230 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:19.565633 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.565595 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-xks-ca\") pod \"fb8864de-322e-4f94-84a1-9857a8ae5946\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " Oct 30 21:32:19.565804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.565776 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-openshift-ca\") pod \"fb8864de-322e-4f94-84a1-9857a8ae5946\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " Oct 30 21:32:19.565884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.565868 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8864de-322e-4f94-84a1-9857a8ae5946-kube-api-access-2cqpr\" (UniqueName: \"kubernetes.io/projected/fb8864de-322e-4f94-84a1-9857a8ae5946-kube-api-access-2cqpr\") pod \"fb8864de-322e-4f94-84a1-9857a8ae5946\" (UID: \"fb8864de-322e-4f94-84a1-9857a8ae5946\") " Oct 30 21:32:19.566120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.566097 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-openshift-ca" pod "fb8864de-322e-4f94-84a1-9857a8ae5946" (UID: "fb8864de-322e-4f94-84a1-9857a8ae5946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:19.566264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.566250 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:19.567924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.567905 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-xks-ca" pod "fb8864de-322e-4f94-84a1-9857a8ae5946" (UID: "fb8864de-322e-4f94-84a1-9857a8ae5946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:19.568053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.568032 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8864de-322e-4f94-84a1-9857a8ae5946-kube-api-access-2cqpr" pod "fb8864de-322e-4f94-84a1-9857a8ae5946" (UID: "fb8864de-322e-4f94-84a1-9857a8ae5946"). InnerVolumeSpecName "kube-api-access-2cqpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:19.667689 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.667600 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqpr\" (UniqueName: \"kubernetes.io/projected/fb8864de-322e-4f94-84a1-9857a8ae5946-kube-api-access-2cqpr\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:19.667689 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:19.667632 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8864de-322e-4f94-84a1-9857a8ae5946-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:20.384849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.384812 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" Oct 30 21:32:20.385296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.384840 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf" event={"ID":"fb8864de-322e-4f94-84a1-9857a8ae5946","Type":"ContainerDied","Data":"632091f944dc01db0e14fccb797c69b3b603f38c1ff272fb5db81d1271275962"} Oct 30 21:32:20.385296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.384897 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="632091f944dc01db0e14fccb797c69b3b603f38c1ff272fb5db81d1271275962" Oct 30 21:32:20.525887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.525858 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:20.578293 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.578263 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-xks-ca\") pod \"347013c5-51f5-46de-8b5e-3caf3c2848e6\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " Oct 30 21:32:20.578467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.578303 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-openshift-ca\") pod \"347013c5-51f5-46de-8b5e-3caf3c2848e6\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " Oct 30 21:32:20.578467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.578348 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347013c5-51f5-46de-8b5e-3caf3c2848e6-kube-api-access-rvq8f\" (UniqueName: \"kubernetes.io/projected/347013c5-51f5-46de-8b5e-3caf3c2848e6-kube-api-access-rvq8f\") pod \"347013c5-51f5-46de-8b5e-3caf3c2848e6\" (UID: \"347013c5-51f5-46de-8b5e-3caf3c2848e6\") " Oct 30 21:32:20.578795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.578773 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-openshift-ca" pod "347013c5-51f5-46de-8b5e-3caf3c2848e6" (UID: "347013c5-51f5-46de-8b5e-3caf3c2848e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:20.580647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.580614 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-xks-ca" pod "347013c5-51f5-46de-8b5e-3caf3c2848e6" (UID: "347013c5-51f5-46de-8b5e-3caf3c2848e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:20.580745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.580651 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347013c5-51f5-46de-8b5e-3caf3c2848e6-kube-api-access-rvq8f" pod "347013c5-51f5-46de-8b5e-3caf3c2848e6" (UID: "347013c5-51f5-46de-8b5e-3caf3c2848e6"). InnerVolumeSpecName "kube-api-access-rvq8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:20.679926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.679846 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:20.679926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.679876 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347013c5-51f5-46de-8b5e-3caf3c2848e6-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:20.679926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:20.679887 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvq8f\" (UniqueName: \"kubernetes.io/projected/347013c5-51f5-46de-8b5e-3caf3c2848e6-kube-api-access-rvq8f\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:21.391650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:21.391624 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" Oct 30 21:32:21.392061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:21.391656 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n" event={"ID":"347013c5-51f5-46de-8b5e-3caf3c2848e6","Type":"ContainerDied","Data":"3fc151afc1e43d23a0932f384a37a5b8f9db8e042722629ccde5ccd395c0b982"} Oct 30 21:32:21.392061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:21.391701 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc151afc1e43d23a0932f384a37a5b8f9db8e042722629ccde5ccd395c0b982" Oct 30 21:32:30.371880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.371847 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd"] Oct 30 21:32:30.372368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.372354 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fb8864de-322e-4f94-84a1-9857a8ae5946" containerName="revoke-keys" Oct 30 21:32:30.372412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.372370 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8864de-322e-4f94-84a1-9857a8ae5946" containerName="revoke-keys" Oct 30 21:32:30.372412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.372407 4526 cpu_manager.go:414] "containerMap: removing container" podUID="347013c5-51f5-46de-8b5e-3caf3c2848e6" containerName="revoke-keys" Oct 30 21:32:30.372472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.372412 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347013c5-51f5-46de-8b5e-3caf3c2848e6" containerName="revoke-keys" Oct 30 21:32:30.372546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.372512 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="347013c5-51f5-46de-8b5e-3caf3c2848e6" containerName="revoke-keys" Oct 30 21:32:30.372546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.372523 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8864de-322e-4f94-84a1-9857a8ae5946" containerName="revoke-keys" Oct 30 21:32:30.375823 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.375806 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.481924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.481886 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"default-dockercfg-qk7t2\"" Oct 30 21:32:30.481924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.481902 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"kube-root-ca.crt\"" Oct 30 21:32:30.482186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.481951 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"openshift-service-ca.crt\"" Oct 30 21:32:30.482186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.482059 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"premium-w0-1b5b5f-kserve-self-signed-certs\"" Oct 30 21:32:30.483578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.483552 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4z88\" (UniqueName: \"kubernetes.io/projected/e631b930-be02-42a4-949a-a74e9b884a24-kube-api-access-t4z88\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.483680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.483582 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-tmp-dir\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.483680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.483631 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-model-cache\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.483749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.483709 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-home\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.483783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.483767 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-dshm\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.483853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.483841 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e631b930-be02-42a4-949a-a74e9b884a24-tls-certs\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.484549 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.484531 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd"] Oct 30 21:32:30.584877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.584841 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-model-cache\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.584877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.584881 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-home\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.585118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.584899 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-dshm\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.585118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.585027 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e631b930-be02-42a4-949a-a74e9b884a24-tls-certs\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.585206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.585189 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4z88\" (UniqueName: \"kubernetes.io/projected/e631b930-be02-42a4-949a-a74e9b884a24-kube-api-access-t4z88\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.585248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.585224 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-tmp-dir\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.585324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.585303 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-model-cache\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.585372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.585324 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-home\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.585627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.585609 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-tmp-dir\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.587402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.587384 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-dshm\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.587577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.587561 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e631b930-be02-42a4-949a-a74e9b884a24-tls-certs\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.645941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.645858 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4z88\" (UniqueName: \"kubernetes.io/projected/e631b930-be02-42a4-949a-a74e9b884a24-kube-api-access-t4z88\") pod \"premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:30.687813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:30.687777 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:31.067544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:31.067519 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd"] Oct 30 21:32:31.069678 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:32:31.069644 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode631b930_be02_42a4_949a_a74e9b884a24.slice/crio-dacdd44358c1e177d011613069ce556625634f4ea65a0fa3d420cd92494b368d WatchSource:0}: Error finding container dacdd44358c1e177d011613069ce556625634f4ea65a0fa3d420cd92494b368d: Status 404 returned error can't find the container with id dacdd44358c1e177d011613069ce556625634f4ea65a0fa3d420cd92494b368d Oct 30 21:32:31.450958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:31.450823 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" event={"ID":"e631b930-be02-42a4-949a-a74e9b884a24","Type":"ContainerStarted","Data":"4cf1bad8af77ba3f2d61651848973d575b3c365b61a2694f077975de4e98c7d0"} Oct 30 21:32:31.450958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:31.450874 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" event={"ID":"e631b930-be02-42a4-949a-a74e9b884a24","Type":"ContainerStarted","Data":"dacdd44358c1e177d011613069ce556625634f4ea65a0fa3d420cd92494b368d"} Oct 30 21:32:31.568368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:31.568306 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podStartSLOduration=1.5682905219999999 podStartE2EDuration="1.568290522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:31.565128677 +0000 UTC m=+941.698501875" watchObservedRunningTime="2026-10-30 21:32:31.568290522 +0000 UTC m=+941.701663720" Oct 30 21:32:31.688187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:31.688152 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:33.120307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:33.120271 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh"] Oct 30 21:32:33.120765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:33.120556 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" containerID="cri-o://794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f" gracePeriod=30 Oct 30 21:32:33.170751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:33.170707 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64"] Oct 30 21:32:33.170990 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:33.170947 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" containerID="cri-o://95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88" gracePeriod=30 Oct 30 21:32:38.436717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.436687 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:32:38.471309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.471283 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:32:38.492745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.492711 4526 generic.go:356] "Generic (PLEG): container finished" podID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerID="794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f" exitCode=0 Oct 30 21:32:38.492943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.492761 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" Oct 30 21:32:38.492943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.492803 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" event={"ID":"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a","Type":"ContainerDied","Data":"794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f"} Oct 30 21:32:38.492943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.492861 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh" event={"ID":"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a","Type":"ContainerDied","Data":"4871e50e7fb0c53efb540826b26994b70c14fac4610bf49edc966a393be6b20e"} Oct 30 21:32:38.492943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.492895 4526 scope.go:122] "RemoveContainer" containerID="794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f" Oct 30 21:32:38.494155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.494138 4526 generic.go:356] "Generic (PLEG): container finished" podID="66449278-1a38-4a04-a93d-1fe119de3c81" containerID="95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88" exitCode=0 Oct 30 21:32:38.494236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.494172 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" event={"ID":"66449278-1a38-4a04-a93d-1fe119de3c81","Type":"ContainerDied","Data":"95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88"} Oct 30 21:32:38.494236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.494183 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" Oct 30 21:32:38.494236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.494206 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64" event={"ID":"66449278-1a38-4a04-a93d-1fe119de3c81","Type":"ContainerDied","Data":"e1bf3111411a892b0135b5a584548f5885fbdd80c4f55cd8af5b022cbc81fbe9"} Oct 30 21:32:38.502015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.501995 4526 scope.go:122] "RemoveContainer" containerID="794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f" Oct 30 21:32:38.502336 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:32:38.502318 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f\": container with ID starting with 794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f not found: ID does not exist" containerID="794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f" Oct 30 21:32:38.502388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.502347 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f"} err="failed to get container status \"794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f\": rpc error: code = NotFound desc = could not find container \"794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f\": container with ID starting with 794cfd6fa36ce33634e32fabba2fe4524aa84710c15735ec3cc4dd34389ed42f not found: ID does not exist" Oct 30 21:32:38.502388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.502367 4526 scope.go:122] "RemoveContainer" containerID="95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88" Oct 30 21:32:38.511162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.511144 4526 scope.go:122] "RemoveContainer" containerID="95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88" Oct 30 21:32:38.511462 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:32:38.511444 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88\": container with ID starting with 95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88 not found: ID does not exist" containerID="95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88" Oct 30 21:32:38.511550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.511475 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88"} err="failed to get container status \"95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88\": rpc error: code = NotFound desc = could not find container \"95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88\": container with ID starting with 95ff0eed26b2520f3d74ad738504eba834db3ca94a170d04337984a20089ce88 not found: ID does not exist" Oct 30 21:32:38.576830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.576791 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-envoy\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.576830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.576835 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-podinfo\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.577121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.576869 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istiod-ca-cert\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.577121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.576902 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-istio-token\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-istio-token\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.577121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577061 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-credential-socket\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.577121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577107 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66449278-1a38-4a04-a93d-1fe119de3c81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66449278-1a38-4a04-a93d-1fe119de3c81-istio-podinfo\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.577330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577130 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-certs\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.577330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577165 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-kube-api-access-gj4qs\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-kube-api-access-gj4qs\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.577330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577203 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-token\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-token\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.577330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577254 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66449278-1a38-4a04-a93d-1fe119de3c81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66449278-1a38-4a04-a93d-1fe119de3c81-istiod-ca-cert\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.577330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577316 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-kube-api-access-jb64d\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-kube-api-access-jb64d\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.577636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577366 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-credential-socket" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.577636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577388 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-data\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.577636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577397 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istiod-ca-cert" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:38.577636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577473 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-data\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.577636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577537 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-socket\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.577636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577584 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-envoy\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.577636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577617 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-certs" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.577636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577629 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-credential-socket\") pod \"66449278-1a38-4a04-a93d-1fe119de3c81\" (UID: \"66449278-1a38-4a04-a93d-1fe119de3c81\") " Oct 30 21:32:38.578035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577667 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-socket\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.578035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577748 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-certs\") pod \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\" (UID: \"1ca52af0-9cb4-4353-98b1-92e69b6c5b9a\") " Oct 30 21:32:38.578035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.577945 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66449278-1a38-4a04-a93d-1fe119de3c81-istiod-ca-cert" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:38.578191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578152 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-certs" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.578191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578156 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-credential-socket" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.578462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578345 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-data" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.578462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578359 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-socket" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.578462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578436 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.578462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578457 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.578684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578471 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.578684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578501 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.578684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578516 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66449278-1a38-4a04-a93d-1fe119de3c81-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.578684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578529 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.578684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578547 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-socket" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.578684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.578640 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-data" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.579960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.579926 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-podinfo" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:38.580087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.579966 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-envoy" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.580545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.580521 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-token" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:38.580657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.580570 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-istio-token" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:38.580899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.580878 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-kube-api-access-jb64d" pod "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" (UID: "1ca52af0-9cb4-4353-98b1-92e69b6c5b9a"). InnerVolumeSpecName "kube-api-access-jb64d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:38.581109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.581087 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-kube-api-access-gj4qs" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "kube-api-access-gj4qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:38.581181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.581128 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-envoy" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.581216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.581168 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66449278-1a38-4a04-a93d-1fe119de3c81-istio-podinfo" pod "66449278-1a38-4a04-a93d-1fe119de3c81" (UID: "66449278-1a38-4a04-a93d-1fe119de3c81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:38.680237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680141 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680176 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680188 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680198 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66449278-1a38-4a04-a93d-1fe119de3c81-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680207 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj4qs\" (UniqueName: \"kubernetes.io/projected/66449278-1a38-4a04-a93d-1fe119de3c81-kube-api-access-gj4qs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680216 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680226 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb64d\" (UniqueName: \"kubernetes.io/projected/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-kube-api-access-jb64d\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680237 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680251 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680259 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680268 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66449278-1a38-4a04-a93d-1fe119de3c81-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.680649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.680277 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.916921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.916876 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh"] Oct 30 21:32:38.920269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:38.920240 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bz6dh"] Oct 30 21:32:39.029229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:39.029136 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64"] Oct 30 21:32:39.033310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:39.033279 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-flf64"] Oct 30 21:32:39.984424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:39.984390 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" path="/var/lib/kubelet/pods/1ca52af0-9cb4-4353-98b1-92e69b6c5b9a/volumes" Oct 30 21:32:39.985049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:39.985029 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" path="/var/lib/kubelet/pods/66449278-1a38-4a04-a93d-1fe119de3c81/volumes" Oct 30 21:32:40.688424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:40.688382 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:40.701639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:40.701607 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:41.524662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.524637 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:32:41.848056 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.848021 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x"] Oct 30 21:32:41.848643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.848618 4526 cpu_manager.go:414] "containerMap: removing container" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" Oct 30 21:32:41.848708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.848644 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" Oct 30 21:32:41.848708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.848658 4526 cpu_manager.go:414] "containerMap: removing container" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" Oct 30 21:32:41.848708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.848663 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" Oct 30 21:32:41.848821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.848746 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="66449278-1a38-4a04-a93d-1fe119de3c81" containerName="istio-proxy" Oct 30 21:32:41.848821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.848756 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca52af0-9cb4-4353-98b1-92e69b6c5b9a" containerName="istio-proxy" Oct 30 21:32:41.853562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.853546 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:41.906901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.906865 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:32:41.906901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.906904 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:32:41.907169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.906924 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:32:41.907228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.907206 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:32:41.909908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:41.909881 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x"] Oct 30 21:32:42.016556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.016519 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.016556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.016559 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz99r\" (UniqueName: \"kubernetes.io/projected/c9b967e8-1484-48c9-b52a-f8e604df8576-kube-api-access-cz99r\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.016850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.016683 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.117994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.117899 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.117994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.117963 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz99r\" (UniqueName: \"kubernetes.io/projected/c9b967e8-1484-48c9-b52a-f8e604df8576-kube-api-access-cz99r\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.118206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.118008 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.118614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.118591 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.120511 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.120476 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.174400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.174365 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz99r\" (UniqueName: \"kubernetes.io/projected/c9b967e8-1484-48c9-b52a-f8e604df8576-kube-api-access-cz99r\") pod \"maas-api-revoke-sub-ef5dc42e-mwq6x\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.467667 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.467589 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:42.479527 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:32:42.479476 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:32:42.844451 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:42.844424 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x"] Oct 30 21:32:42.846594 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:32:42.846557 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b967e8_1484_48c9_b52a_f8e604df8576.slice/crio-5f46a8c1c2bf7dd7c9bf058ace5e9e2ae68e94ded0793e02a9aa132789304591 WatchSource:0}: Error finding container 5f46a8c1c2bf7dd7c9bf058ace5e9e2ae68e94ded0793e02a9aa132789304591: Status 404 returned error can't find the container with id 5f46a8c1c2bf7dd7c9bf058ace5e9e2ae68e94ded0793e02a9aa132789304591 Oct 30 21:32:43.530174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:43.530077 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" event={"ID":"c9b967e8-1484-48c9-b52a-f8e604df8576","Type":"ContainerStarted","Data":"5fd284b67bd274c2a3f409f528ca3a59ee892362321347712531986c3bda2032"} Oct 30 21:32:43.530174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:43.530135 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" event={"ID":"c9b967e8-1484-48c9-b52a-f8e604df8576","Type":"ContainerStarted","Data":"5f46a8c1c2bf7dd7c9bf058ace5e9e2ae68e94ded0793e02a9aa132789304591"} Oct 30 21:32:43.643139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:43.643073 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" podStartSLOduration=2.643053127 podStartE2EDuration="2.643053127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:43.64208409 +0000 UTC m=+953.775457266" watchObservedRunningTime="2026-10-30 21:32:43.643053127 +0000 UTC m=+953.776426326" Oct 30 21:32:44.536838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:44.536808 4526 generic.go:356] "Generic (PLEG): container finished" podID="c9b967e8-1484-48c9-b52a-f8e604df8576" containerID="5fd284b67bd274c2a3f409f528ca3a59ee892362321347712531986c3bda2032" exitCode=0 Oct 30 21:32:44.537327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:44.536919 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" event={"ID":"c9b967e8-1484-48c9-b52a-f8e604df8576","Type":"ContainerDied","Data":"5fd284b67bd274c2a3f409f528ca3a59ee892362321347712531986c3bda2032"} Oct 30 21:32:45.681865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.681834 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:45.857448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.857368 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-xks-ca\") pod \"c9b967e8-1484-48c9-b52a-f8e604df8576\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " Oct 30 21:32:45.857448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.857404 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b967e8-1484-48c9-b52a-f8e604df8576-kube-api-access-cz99r\" (UniqueName: \"kubernetes.io/projected/c9b967e8-1484-48c9-b52a-f8e604df8576-kube-api-access-cz99r\") pod \"c9b967e8-1484-48c9-b52a-f8e604df8576\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " Oct 30 21:32:45.857682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.857586 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-openshift-ca\") pod \"c9b967e8-1484-48c9-b52a-f8e604df8576\" (UID: \"c9b967e8-1484-48c9-b52a-f8e604df8576\") " Oct 30 21:32:45.857939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.857915 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-openshift-ca" pod "c9b967e8-1484-48c9-b52a-f8e604df8576" (UID: "c9b967e8-1484-48c9-b52a-f8e604df8576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:45.858016 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.858001 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.859715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.859693 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-xks-ca" pod "c9b967e8-1484-48c9-b52a-f8e604df8576" (UID: "c9b967e8-1484-48c9-b52a-f8e604df8576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:45.859715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.859700 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b967e8-1484-48c9-b52a-f8e604df8576-kube-api-access-cz99r" pod "c9b967e8-1484-48c9-b52a-f8e604df8576" (UID: "c9b967e8-1484-48c9-b52a-f8e604df8576"). InnerVolumeSpecName "kube-api-access-cz99r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:45.959559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.959517 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b967e8-1484-48c9-b52a-f8e604df8576-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.959559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:45.959556 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz99r\" (UniqueName: \"kubernetes.io/projected/c9b967e8-1484-48c9-b52a-f8e604df8576-kube-api-access-cz99r\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:46.550132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:46.550104 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" Oct 30 21:32:46.550316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:46.550103 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x" event={"ID":"c9b967e8-1484-48c9-b52a-f8e604df8576","Type":"ContainerDied","Data":"5f46a8c1c2bf7dd7c9bf058ace5e9e2ae68e94ded0793e02a9aa132789304591"} Oct 30 21:32:46.550316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:46.550227 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f46a8c1c2bf7dd7c9bf058ace5e9e2ae68e94ded0793e02a9aa132789304591" Oct 30 21:32:51.528573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.528540 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7"] Oct 30 21:32:51.529052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.529038 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c9b967e8-1484-48c9-b52a-f8e604df8576" containerName="revoke-keys" Oct 30 21:32:51.529100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.529053 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b967e8-1484-48c9-b52a-f8e604df8576" containerName="revoke-keys" Oct 30 21:32:51.529158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.529148 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b967e8-1484-48c9-b52a-f8e604df8576" containerName="revoke-keys" Oct 30 21:32:51.533635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.533618 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.615479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.615424 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb4hz\" (UniqueName: \"kubernetes.io/projected/93671712-15ba-4b9a-a476-a97285a2268d-kube-api-access-qb4hz\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.615674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.615518 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.615744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.615715 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.634886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.634855 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:32:51.635046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.634902 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:32:51.635046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.634925 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:32:51.635046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.634986 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:32:51.637698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.637677 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7"] Oct 30 21:32:51.717102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.717060 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb4hz\" (UniqueName: \"kubernetes.io/projected/93671712-15ba-4b9a-a476-a97285a2268d-kube-api-access-qb4hz\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.717102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.717119 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.717379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.717192 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.717838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.717819 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.719727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.719707 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.776361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.776334 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb4hz\" (UniqueName: \"kubernetes.io/projected/93671712-15ba-4b9a-a476-a97285a2268d-kube-api-access-qb4hz\") pod \"maas-api-revoke-sub-86170231-kgvv7\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:51.847231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:51.847198 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:52.225061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.225027 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7"] Oct 30 21:32:52.228164 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:32:52.228140 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93671712_15ba_4b9a_a476_a97285a2268d.slice/crio-1d73dfb1a8ddade57715594d0168941c92489e07fd908ca32d538e7abafaa5df WatchSource:0}: Error finding container 1d73dfb1a8ddade57715594d0168941c92489e07fd908ca32d538e7abafaa5df: Status 404 returned error can't find the container with id 1d73dfb1a8ddade57715594d0168941c92489e07fd908ca32d538e7abafaa5df Oct 30 21:32:52.383295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.383261 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj"] Oct 30 21:32:52.388264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.388245 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.426006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.425972 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-home\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.426170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.426031 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-dshm\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.426170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.426149 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-model-cache\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.426248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.426218 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700e8b87-dae4-4881-a139-a090e954220c-tls-certs\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.426309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.426288 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-tmp-dir\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.426396 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.426383 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xjx\" (UniqueName: \"kubernetes.io/projected/700e8b87-dae4-4881-a139-a090e954220c-kube-api-access-b8xjx\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.441519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.441454 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"premium-w6-89c0fd-kserve-self-signed-certs\"" Oct 30 21:32:52.444843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.444814 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj"] Oct 30 21:32:52.527055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527023 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700e8b87-dae4-4881-a139-a090e954220c-tls-certs\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.527218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527065 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-tmp-dir\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.527218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527111 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xjx\" (UniqueName: \"kubernetes.io/projected/700e8b87-dae4-4881-a139-a090e954220c-kube-api-access-b8xjx\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.527218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527158 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-home\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.527218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527189 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-dshm\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.527448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527239 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-model-cache\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.527587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527567 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-tmp-dir\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.527637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527582 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-home\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.527671 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.527647 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-model-cache\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.529482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.529463 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-dshm\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.529916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.529895 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700e8b87-dae4-4881-a139-a090e954220c-tls-certs\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.583929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.583895 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xjx\" (UniqueName: \"kubernetes.io/projected/700e8b87-dae4-4881-a139-a090e954220c-kube-api-access-b8xjx\") pod \"premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.589286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.589248 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" event={"ID":"93671712-15ba-4b9a-a476-a97285a2268d","Type":"ContainerStarted","Data":"ea4401ac789f3408cc253dce6ca2898ff25822cc2f300212d41dddf29f7a029e"} Oct 30 21:32:52.589474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.589303 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" event={"ID":"93671712-15ba-4b9a-a476-a97285a2268d","Type":"ContainerStarted","Data":"1d73dfb1a8ddade57715594d0168941c92489e07fd908ca32d538e7abafaa5df"} Oct 30 21:32:52.701921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.701890 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:52.703559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:52.703356 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" podStartSLOduration=1.703339004 podStartE2EDuration="1.703339004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:52.702482643 +0000 UTC m=+962.835855835" watchObservedRunningTime="2026-10-30 21:32:52.703339004 +0000 UTC m=+962.836712203" Oct 30 21:32:53.081979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:53.081950 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj"] Oct 30 21:32:53.113232 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:32:53.113192 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod700e8b87_dae4_4881_a139_a090e954220c.slice/crio-0868b2409c2cc8013d4603811d81b40b34f8740c34d26069ccb8856dcd185a1a WatchSource:0}: Error finding container 0868b2409c2cc8013d4603811d81b40b34f8740c34d26069ccb8856dcd185a1a: Status 404 returned error can't find the container with id 0868b2409c2cc8013d4603811d81b40b34f8740c34d26069ccb8856dcd185a1a Oct 30 21:32:53.594340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:53.594307 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" event={"ID":"700e8b87-dae4-4881-a139-a090e954220c","Type":"ContainerStarted","Data":"ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f"} Oct 30 21:32:53.594865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:53.594350 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" event={"ID":"700e8b87-dae4-4881-a139-a090e954220c","Type":"ContainerStarted","Data":"0868b2409c2cc8013d4603811d81b40b34f8740c34d26069ccb8856dcd185a1a"} Oct 30 21:32:53.595902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:53.595879 4526 generic.go:356] "Generic (PLEG): container finished" podID="93671712-15ba-4b9a-a476-a97285a2268d" containerID="ea4401ac789f3408cc253dce6ca2898ff25822cc2f300212d41dddf29f7a029e" exitCode=0 Oct 30 21:32:53.595984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:53.595937 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" event={"ID":"93671712-15ba-4b9a-a476-a97285a2268d","Type":"ContainerDied","Data":"ea4401ac789f3408cc253dce6ca2898ff25822cc2f300212d41dddf29f7a029e"} Oct 30 21:32:53.705698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:53.702962 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:32:53.711939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:53.711845 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podStartSLOduration=1.711825243 podStartE2EDuration="1.711825243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:53.708288115 +0000 UTC m=+963.841661313" watchObservedRunningTime="2026-10-30 21:32:53.711825243 +0000 UTC m=+963.845198445" Oct 30 21:32:54.751039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.751012 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:54.857387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.857309 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-openshift-ca\") pod \"93671712-15ba-4b9a-a476-a97285a2268d\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " Oct 30 21:32:54.857387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.857365 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93671712-15ba-4b9a-a476-a97285a2268d-kube-api-access-qb4hz\" (UniqueName: \"kubernetes.io/projected/93671712-15ba-4b9a-a476-a97285a2268d-kube-api-access-qb4hz\") pod \"93671712-15ba-4b9a-a476-a97285a2268d\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " Oct 30 21:32:54.857646 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.857531 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-xks-ca\") pod \"93671712-15ba-4b9a-a476-a97285a2268d\" (UID: \"93671712-15ba-4b9a-a476-a97285a2268d\") " Oct 30 21:32:54.857767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.857740 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-openshift-ca" pod "93671712-15ba-4b9a-a476-a97285a2268d" (UID: "93671712-15ba-4b9a-a476-a97285a2268d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:54.859724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.859697 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93671712-15ba-4b9a-a476-a97285a2268d-kube-api-access-qb4hz" pod "93671712-15ba-4b9a-a476-a97285a2268d" (UID: "93671712-15ba-4b9a-a476-a97285a2268d"). InnerVolumeSpecName "kube-api-access-qb4hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:54.859724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.859699 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-xks-ca" pod "93671712-15ba-4b9a-a476-a97285a2268d" (UID: "93671712-15ba-4b9a-a476-a97285a2268d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:54.958975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.958940 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:54.958975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.958969 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93671712-15ba-4b9a-a476-a97285a2268d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:54.958975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:54.958980 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb4hz\" (UniqueName: \"kubernetes.io/projected/93671712-15ba-4b9a-a476-a97285a2268d-kube-api-access-qb4hz\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:32:55.607758 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:55.607732 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" Oct 30 21:32:55.607914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:55.607774 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7" event={"ID":"93671712-15ba-4b9a-a476-a97285a2268d","Type":"ContainerDied","Data":"1d73dfb1a8ddade57715594d0168941c92489e07fd908ca32d538e7abafaa5df"} Oct 30 21:32:55.607914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:32:55.607818 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d73dfb1a8ddade57715594d0168941c92489e07fd908ca32d538e7abafaa5df" Oct 30 21:33:02.702582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:02.702539 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:33:02.715709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:02.715684 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:33:03.663016 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:03.662986 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:33:09.140413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.140381 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp"] Oct 30 21:33:09.140939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.140924 4526 cpu_manager.go:414] "containerMap: removing container" podUID="93671712-15ba-4b9a-a476-a97285a2268d" containerName="revoke-keys" Oct 30 21:33:09.140982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.140941 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93671712-15ba-4b9a-a476-a97285a2268d" containerName="revoke-keys" Oct 30 21:33:09.141032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.141023 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="93671712-15ba-4b9a-a476-a97285a2268d" containerName="revoke-keys" Oct 30 21:33:09.144172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.144155 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.197972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.197942 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:09.216187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.216150 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.216375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.216259 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76kh5\" (UniqueName: \"kubernetes.io/projected/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-kube-api-access-76kh5\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.216375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.216332 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.247680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.247639 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:33:09.247680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.247639 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:33:09.248022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.247751 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:33:09.250665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.250635 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp"] Oct 30 21:33:09.317867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.317837 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.318092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.317902 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kh5\" (UniqueName: \"kubernetes.io/projected/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-kube-api-access-76kh5\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.318092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.317974 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.318596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.318573 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.320476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.320458 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.375455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.375416 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kh5\" (UniqueName: \"kubernetes.io/projected/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-kube-api-access-76kh5\") pod \"maas-api-revoke-sub-f99a2209-59rhp\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.458425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.458329 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:09.833777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:09.833751 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp"] Oct 30 21:33:09.835474 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:09.835451 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8251fafa_ad43_4470_b1ba_6f8849ccc8ff.slice/crio-d4ac4b4d02fafb1b4ad3990f45309d57570ed8ba9fd0a10e860b03d515e07b52 WatchSource:0}: Error finding container d4ac4b4d02fafb1b4ad3990f45309d57570ed8ba9fd0a10e860b03d515e07b52: Status 404 returned error can't find the container with id d4ac4b4d02fafb1b4ad3990f45309d57570ed8ba9fd0a10e860b03d515e07b52 Oct 30 21:33:10.692723 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:10.692674 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" event={"ID":"8251fafa-ad43-4470-b1ba-6f8849ccc8ff","Type":"ContainerStarted","Data":"e4591bfd7dc7967a6aa27427cccf083645a5e8255d68491c9dd63151e662c2f3"} Oct 30 21:33:10.693122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:10.692751 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" event={"ID":"8251fafa-ad43-4470-b1ba-6f8849ccc8ff","Type":"ContainerStarted","Data":"d4ac4b4d02fafb1b4ad3990f45309d57570ed8ba9fd0a10e860b03d515e07b52"} Oct 30 21:33:10.806192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:10.806133 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" podStartSLOduration=1.806116037 podStartE2EDuration="1.806116037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:10.805425891 +0000 UTC m=+980.938799091" watchObservedRunningTime="2026-10-30 21:33:10.806116037 +0000 UTC m=+980.939489235" Oct 30 21:33:11.699888 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:11.699855 4526 generic.go:356] "Generic (PLEG): container finished" podID="8251fafa-ad43-4470-b1ba-6f8849ccc8ff" containerID="e4591bfd7dc7967a6aa27427cccf083645a5e8255d68491c9dd63151e662c2f3" exitCode=0 Oct 30 21:33:11.700340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:11.699938 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" event={"ID":"8251fafa-ad43-4470-b1ba-6f8849ccc8ff","Type":"ContainerDied","Data":"e4591bfd7dc7967a6aa27427cccf083645a5e8255d68491c9dd63151e662c2f3"} Oct 30 21:33:11.959532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:11.959430 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh"] Oct 30 21:33:11.963266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:11.963250 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.020267 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.020235 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh"] Oct 30 21:33:12.049355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.049316 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbp5m\" (UniqueName: \"kubernetes.io/projected/99fc9fcb-2145-4938-a926-ab6742153162-kube-api-access-kbp5m\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.049562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.049374 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.049562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.049512 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.150349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.150320 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbp5m\" (UniqueName: \"kubernetes.io/projected/99fc9fcb-2145-4938-a926-ab6742153162-kube-api-access-kbp5m\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.150560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.150392 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.150560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.150421 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.151090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.151068 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.153216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.153195 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.207013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.206975 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbp5m\" (UniqueName: \"kubernetes.io/projected/99fc9fcb-2145-4938-a926-ab6742153162-kube-api-access-kbp5m\") pod \"maas-api-revoke-sub-460290b5-gk5xh\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.279268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.279185 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:12.478906 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:33:12.478870 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:33:12.652871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.652843 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh"] Oct 30 21:33:12.654638 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:12.654611 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fc9fcb_2145_4938_a926_ab6742153162.slice/crio-b0e4ce872d8a616dad79f6ad9129d6d9dd1ae1568154848e5b009749d865e614 WatchSource:0}: Error finding container b0e4ce872d8a616dad79f6ad9129d6d9dd1ae1568154848e5b009749d865e614: Status 404 returned error can't find the container with id b0e4ce872d8a616dad79f6ad9129d6d9dd1ae1568154848e5b009749d865e614 Oct 30 21:33:12.707766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.707736 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" event={"ID":"99fc9fcb-2145-4938-a926-ab6742153162","Type":"ContainerStarted","Data":"b0e4ce872d8a616dad79f6ad9129d6d9dd1ae1568154848e5b009749d865e614"} Oct 30 21:33:12.837404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.837378 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:12.959677 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.959644 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-xks-ca\") pod \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " Oct 30 21:33:12.959859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.959819 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-openshift-ca\") pod \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " Oct 30 21:33:12.959963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.959948 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-kube-api-access-76kh5\" (UniqueName: \"kubernetes.io/projected/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-kube-api-access-76kh5\") pod \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\" (UID: \"8251fafa-ad43-4470-b1ba-6f8849ccc8ff\") " Oct 30 21:33:12.960208 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.960180 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-openshift-ca" pod "8251fafa-ad43-4470-b1ba-6f8849ccc8ff" (UID: "8251fafa-ad43-4470-b1ba-6f8849ccc8ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:12.960602 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.960386 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:12.962082 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.962059 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-xks-ca" pod "8251fafa-ad43-4470-b1ba-6f8849ccc8ff" (UID: "8251fafa-ad43-4470-b1ba-6f8849ccc8ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:12.962207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:12.962156 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-kube-api-access-76kh5" pod "8251fafa-ad43-4470-b1ba-6f8849ccc8ff" (UID: "8251fafa-ad43-4470-b1ba-6f8849ccc8ff"). InnerVolumeSpecName "kube-api-access-76kh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:13.061536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:13.061505 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:13.061536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:13.061534 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76kh5\" (UniqueName: \"kubernetes.io/projected/8251fafa-ad43-4470-b1ba-6f8849ccc8ff-kube-api-access-76kh5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:13.714790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:13.714763 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" Oct 30 21:33:13.714790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:13.714776 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp" event={"ID":"8251fafa-ad43-4470-b1ba-6f8849ccc8ff","Type":"ContainerDied","Data":"d4ac4b4d02fafb1b4ad3990f45309d57570ed8ba9fd0a10e860b03d515e07b52"} Oct 30 21:33:13.715303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:13.714847 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ac4b4d02fafb1b4ad3990f45309d57570ed8ba9fd0a10e860b03d515e07b52" Oct 30 21:33:13.716420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:13.716391 4526 generic.go:356] "Generic (PLEG): container finished" podID="99fc9fcb-2145-4938-a926-ab6742153162" containerID="0f53de66fd17c8b508896c754705d57dcf45db5dc905fcc8d9223380595eede9" exitCode=0 Oct 30 21:33:13.716577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:13.716433 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" event={"ID":"99fc9fcb-2145-4938-a926-ab6742153162","Type":"ContainerDied","Data":"0f53de66fd17c8b508896c754705d57dcf45db5dc905fcc8d9223380595eede9"} Oct 30 21:33:14.869978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:14.869949 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:14.982508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:14.982418 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99fc9fcb-2145-4938-a926-ab6742153162-kube-api-access-kbp5m\" (UniqueName: \"kubernetes.io/projected/99fc9fcb-2145-4938-a926-ab6742153162-kube-api-access-kbp5m\") pod \"99fc9fcb-2145-4938-a926-ab6742153162\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " Oct 30 21:33:14.982508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:14.982468 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-xks-ca\") pod \"99fc9fcb-2145-4938-a926-ab6742153162\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " Oct 30 21:33:14.982759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:14.982584 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-openshift-ca\") pod \"99fc9fcb-2145-4938-a926-ab6742153162\" (UID: \"99fc9fcb-2145-4938-a926-ab6742153162\") " Oct 30 21:33:14.983093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:14.983046 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-openshift-ca" pod "99fc9fcb-2145-4938-a926-ab6742153162" (UID: "99fc9fcb-2145-4938-a926-ab6742153162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:14.985241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:14.985210 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-xks-ca" pod "99fc9fcb-2145-4938-a926-ab6742153162" (UID: "99fc9fcb-2145-4938-a926-ab6742153162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:14.985830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:14.985809 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fc9fcb-2145-4938-a926-ab6742153162-kube-api-access-kbp5m" pod "99fc9fcb-2145-4938-a926-ab6742153162" (UID: "99fc9fcb-2145-4938-a926-ab6742153162"). InnerVolumeSpecName "kube-api-access-kbp5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:15.084514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:15.084460 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbp5m\" (UniqueName: \"kubernetes.io/projected/99fc9fcb-2145-4938-a926-ab6742153162-kube-api-access-kbp5m\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:15.084705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:15.084527 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:15.084705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:15.084543 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fc9fcb-2145-4938-a926-ab6742153162-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:15.729363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:15.729324 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" event={"ID":"99fc9fcb-2145-4938-a926-ab6742153162","Type":"ContainerDied","Data":"b0e4ce872d8a616dad79f6ad9129d6d9dd1ae1568154848e5b009749d865e614"} Oct 30 21:33:15.729363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:15.729356 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh" Oct 30 21:33:15.729629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:15.729381 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0e4ce872d8a616dad79f6ad9129d6d9dd1ae1568154848e5b009749d865e614" Oct 30 21:33:22.742855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.742811 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5"] Oct 30 21:33:22.743359 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.743344 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8251fafa-ad43-4470-b1ba-6f8849ccc8ff" containerName="revoke-keys" Oct 30 21:33:22.743434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.743361 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8251fafa-ad43-4470-b1ba-6f8849ccc8ff" containerName="revoke-keys" Oct 30 21:33:22.743434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.743389 4526 cpu_manager.go:414] "containerMap: removing container" podUID="99fc9fcb-2145-4938-a926-ab6742153162" containerName="revoke-keys" Oct 30 21:33:22.743434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.743395 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fc9fcb-2145-4938-a926-ab6742153162" containerName="revoke-keys" Oct 30 21:33:22.743552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.743521 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fc9fcb-2145-4938-a926-ab6742153162" containerName="revoke-keys" Oct 30 21:33:22.743552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.743541 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8251fafa-ad43-4470-b1ba-6f8849ccc8ff" containerName="revoke-keys" Oct 30 21:33:22.747013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.746998 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.762466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.762445 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8f26\" (UniqueName: \"kubernetes.io/projected/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-kube-api-access-t8f26\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.762602 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.762583 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.762648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.762629 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.799510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.799454 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:33:22.848642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.848603 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:22.848642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.848624 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:33:22.849184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.849133 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:33:22.855150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.855125 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5"] Oct 30 21:33:22.864058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.864032 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8f26\" (UniqueName: \"kubernetes.io/projected/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-kube-api-access-t8f26\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.864185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.864145 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.864247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.864182 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.864928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.864855 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.866575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.866557 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:22.920606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:22.920583 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8f26\" (UniqueName: \"kubernetes.io/projected/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-kube-api-access-t8f26\") pod \"maas-api-revoke-sub-c1f35186-9vfx5\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:23.060257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.060228 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:23.307572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.307537 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt"] Oct 30 21:33:23.312456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.312401 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.369894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.369861 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt"] Oct 30 21:33:23.370458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.370437 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.370622 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.370480 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtzr\" (UniqueName: \"kubernetes.io/projected/f36a4d63-9c35-4824-9cf1-77c96d325cac-kube-api-access-7rtzr\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.370622 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.370531 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.471382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.471344 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtzr\" (UniqueName: \"kubernetes.io/projected/f36a4d63-9c35-4824-9cf1-77c96d325cac-kube-api-access-7rtzr\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.471579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.471402 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.471630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.471592 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.472059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.472042 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.474023 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.473997 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.483328 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.483302 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5"] Oct 30 21:33:23.485657 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:23.485631 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dd9c35_5a0d_42ea_a30f_633a2c95fe3c.slice/crio-502022cc142250aa5e87f2f4e9b4a9b3506531d92da40f37491eed648d6cc265 WatchSource:0}: Error finding container 502022cc142250aa5e87f2f4e9b4a9b3506531d92da40f37491eed648d6cc265: Status 404 returned error can't find the container with id 502022cc142250aa5e87f2f4e9b4a9b3506531d92da40f37491eed648d6cc265 Oct 30 21:33:23.528084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.528061 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtzr\" (UniqueName: \"kubernetes.io/projected/f36a4d63-9c35-4824-9cf1-77c96d325cac-kube-api-access-7rtzr\") pod \"maas-api-revoke-sub-4ee78e7d-b97lt\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.626631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.626556 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:23.773577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.773533 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" event={"ID":"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c","Type":"ContainerStarted","Data":"502022cc142250aa5e87f2f4e9b4a9b3506531d92da40f37491eed648d6cc265"} Oct 30 21:33:23.836419 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.836386 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng"] Oct 30 21:33:23.840179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.840165 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:23.875391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.875360 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:23.875566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.875402 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:23.875644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.875610 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjgbn\" (UniqueName: \"kubernetes.io/projected/a9c5f415-2ae4-419f-8b5d-9147a4a41204-kube-api-access-cjgbn\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:23.897465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.897398 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng"] Oct 30 21:33:23.976270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.976237 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjgbn\" (UniqueName: \"kubernetes.io/projected/a9c5f415-2ae4-419f-8b5d-9147a4a41204-kube-api-access-cjgbn\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:23.976474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.976332 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:23.976474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.976373 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:23.977086 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.977066 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:23.978972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:23.978950 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:24.005309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.005286 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt"] Oct 30 21:33:24.006822 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:24.006799 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36a4d63_9c35_4824_9cf1_77c96d325cac.slice/crio-af11d1178ad446625eb6f8e4e523dac1075121b1f69852fe20a21e6d4a12b148 WatchSource:0}: Error finding container af11d1178ad446625eb6f8e4e523dac1075121b1f69852fe20a21e6d4a12b148: Status 404 returned error can't find the container with id af11d1178ad446625eb6f8e4e523dac1075121b1f69852fe20a21e6d4a12b148 Oct 30 21:33:24.032815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.032791 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjgbn\" (UniqueName: \"kubernetes.io/projected/a9c5f415-2ae4-419f-8b5d-9147a4a41204-kube-api-access-cjgbn\") pod \"maas-api-revoke-sub-fc968c57-2p8ng\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:24.154439 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.154350 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:24.533391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.533358 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng"] Oct 30 21:33:24.535355 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:24.535327 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c5f415_2ae4_419f_8b5d_9147a4a41204.slice/crio-7d0fd166dbb2db18c91afe7a0edb90b465eabc8111fef1c0d5650a03f6c0bc1b WatchSource:0}: Error finding container 7d0fd166dbb2db18c91afe7a0edb90b465eabc8111fef1c0d5650a03f6c0bc1b: Status 404 returned error can't find the container with id 7d0fd166dbb2db18c91afe7a0edb90b465eabc8111fef1c0d5650a03f6c0bc1b Oct 30 21:33:24.788289 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.788252 4526 generic.go:356] "Generic (PLEG): container finished" podID="71dd9c35-5a0d-42ea-a30f-633a2c95fe3c" containerID="1122d9772d02abf3cd9251d2146add48d1e2af62c9185ab8f1e76b86386618c5" exitCode=0 Oct 30 21:33:24.788753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.788333 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" event={"ID":"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c","Type":"ContainerDied","Data":"1122d9772d02abf3cd9251d2146add48d1e2af62c9185ab8f1e76b86386618c5"} Oct 30 21:33:24.789877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.789849 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" event={"ID":"a9c5f415-2ae4-419f-8b5d-9147a4a41204","Type":"ContainerStarted","Data":"7d0fd166dbb2db18c91afe7a0edb90b465eabc8111fef1c0d5650a03f6c0bc1b"} Oct 30 21:33:24.791374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.791353 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" event={"ID":"f36a4d63-9c35-4824-9cf1-77c96d325cac","Type":"ContainerStarted","Data":"20435decf13ffa938bdec57a71b4ac7f85ded45168fefc570f14e39e23cf8f18"} Oct 30 21:33:24.791500 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:24.791388 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" event={"ID":"f36a4d63-9c35-4824-9cf1-77c96d325cac","Type":"ContainerStarted","Data":"af11d1178ad446625eb6f8e4e523dac1075121b1f69852fe20a21e6d4a12b148"} Oct 30 21:33:25.012999 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:25.012939 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" podStartSLOduration=2.012919426 podStartE2EDuration="2.012919426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:25.012094193 +0000 UTC m=+995.145467392" watchObservedRunningTime="2026-10-30 21:33:25.012919426 +0000 UTC m=+995.146292626" Oct 30 21:33:25.798860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:25.798824 4526 generic.go:356] "Generic (PLEG): container finished" podID="a9c5f415-2ae4-419f-8b5d-9147a4a41204" containerID="4b34186f7a6e8a3a52d23650495c69455bfb30573a3b24dff4d3deaf02476adc" exitCode=0 Oct 30 21:33:25.799300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:25.798913 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" event={"ID":"a9c5f415-2ae4-419f-8b5d-9147a4a41204","Type":"ContainerDied","Data":"4b34186f7a6e8a3a52d23650495c69455bfb30573a3b24dff4d3deaf02476adc"} Oct 30 21:33:25.800426 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:25.800405 4526 generic.go:356] "Generic (PLEG): container finished" podID="f36a4d63-9c35-4824-9cf1-77c96d325cac" containerID="20435decf13ffa938bdec57a71b4ac7f85ded45168fefc570f14e39e23cf8f18" exitCode=0 Oct 30 21:33:25.800520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:25.800482 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" event={"ID":"f36a4d63-9c35-4824-9cf1-77c96d325cac","Type":"ContainerDied","Data":"20435decf13ffa938bdec57a71b4ac7f85ded45168fefc570f14e39e23cf8f18"} Oct 30 21:33:25.938482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:25.938457 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:25.999992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:25.999963 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-kube-api-access-t8f26\" (UniqueName: \"kubernetes.io/projected/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-kube-api-access-t8f26\") pod \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " Oct 30 21:33:26.000150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.000007 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-xks-ca\") pod \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " Oct 30 21:33:26.000196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.000170 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-openshift-ca\") pod \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\" (UID: \"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c\") " Oct 30 21:33:26.000536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.000507 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-openshift-ca" pod "71dd9c35-5a0d-42ea-a30f-633a2c95fe3c" (UID: "71dd9c35-5a0d-42ea-a30f-633a2c95fe3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:26.002236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.002217 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-xks-ca" pod "71dd9c35-5a0d-42ea-a30f-633a2c95fe3c" (UID: "71dd9c35-5a0d-42ea-a30f-633a2c95fe3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:26.002299 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.002275 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-kube-api-access-t8f26" pod "71dd9c35-5a0d-42ea-a30f-633a2c95fe3c" (UID: "71dd9c35-5a0d-42ea-a30f-633a2c95fe3c"). InnerVolumeSpecName "kube-api-access-t8f26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:26.101910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.101834 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:26.101910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.101861 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8f26\" (UniqueName: \"kubernetes.io/projected/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-kube-api-access-t8f26\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:26.101910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.101871 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:26.808282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.808256 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" Oct 30 21:33:26.808736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.808293 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5" event={"ID":"71dd9c35-5a0d-42ea-a30f-633a2c95fe3c","Type":"ContainerDied","Data":"502022cc142250aa5e87f2f4e9b4a9b3506531d92da40f37491eed648d6cc265"} Oct 30 21:33:26.808736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.808355 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="502022cc142250aa5e87f2f4e9b4a9b3506531d92da40f37491eed648d6cc265" Oct 30 21:33:26.951074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:26.951050 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:27.001351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.001326 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:27.011920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.011852 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36a4d63-9c35-4824-9cf1-77c96d325cac-kube-api-access-7rtzr\" (UniqueName: \"kubernetes.io/projected/f36a4d63-9c35-4824-9cf1-77c96d325cac-kube-api-access-7rtzr\") pod \"f36a4d63-9c35-4824-9cf1-77c96d325cac\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " Oct 30 21:33:27.012121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.012028 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-xks-ca\") pod \"f36a4d63-9c35-4824-9cf1-77c96d325cac\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " Oct 30 21:33:27.012121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.012090 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-openshift-ca\") pod \"f36a4d63-9c35-4824-9cf1-77c96d325cac\" (UID: \"f36a4d63-9c35-4824-9cf1-77c96d325cac\") " Oct 30 21:33:27.014121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.014096 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-openshift-ca" pod "f36a4d63-9c35-4824-9cf1-77c96d325cac" (UID: "f36a4d63-9c35-4824-9cf1-77c96d325cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:27.015028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.014996 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-xks-ca" pod "f36a4d63-9c35-4824-9cf1-77c96d325cac" (UID: "f36a4d63-9c35-4824-9cf1-77c96d325cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:27.016813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.016761 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36a4d63-9c35-4824-9cf1-77c96d325cac-kube-api-access-7rtzr" pod "f36a4d63-9c35-4824-9cf1-77c96d325cac" (UID: "f36a4d63-9c35-4824-9cf1-77c96d325cac"). InnerVolumeSpecName "kube-api-access-7rtzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:27.113223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.113144 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-openshift-ca\") pod \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " Oct 30 21:33:27.113400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.113245 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-xks-ca\") pod \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " Oct 30 21:33:27.113400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.113303 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c5f415-2ae4-419f-8b5d-9147a4a41204-kube-api-access-cjgbn\" (UniqueName: \"kubernetes.io/projected/a9c5f415-2ae4-419f-8b5d-9147a4a41204-kube-api-access-cjgbn\") pod \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\" (UID: \"a9c5f415-2ae4-419f-8b5d-9147a4a41204\") " Oct 30 21:33:27.113629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.113600 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-openshift-ca" pod "a9c5f415-2ae4-419f-8b5d-9147a4a41204" (UID: "a9c5f415-2ae4-419f-8b5d-9147a4a41204"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:27.113854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.113828 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:27.113854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.113856 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36a4d63-9c35-4824-9cf1-77c96d325cac-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:27.113854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.113872 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:27.113854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.113885 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtzr\" (UniqueName: \"kubernetes.io/projected/f36a4d63-9c35-4824-9cf1-77c96d325cac-kube-api-access-7rtzr\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:27.115594 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.115566 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-xks-ca" pod "a9c5f415-2ae4-419f-8b5d-9147a4a41204" (UID: "a9c5f415-2ae4-419f-8b5d-9147a4a41204"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:27.115679 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.115591 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c5f415-2ae4-419f-8b5d-9147a4a41204-kube-api-access-cjgbn" pod "a9c5f415-2ae4-419f-8b5d-9147a4a41204" (UID: "a9c5f415-2ae4-419f-8b5d-9147a4a41204"). InnerVolumeSpecName "kube-api-access-cjgbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:27.215536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.215482 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c5f415-2ae4-419f-8b5d-9147a4a41204-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:27.215536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.215537 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjgbn\" (UniqueName: \"kubernetes.io/projected/a9c5f415-2ae4-419f-8b5d-9147a4a41204-kube-api-access-cjgbn\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:27.816001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.815975 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" Oct 30 21:33:27.816001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.815982 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng" event={"ID":"a9c5f415-2ae4-419f-8b5d-9147a4a41204","Type":"ContainerDied","Data":"7d0fd166dbb2db18c91afe7a0edb90b465eabc8111fef1c0d5650a03f6c0bc1b"} Oct 30 21:33:27.816602 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.816035 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d0fd166dbb2db18c91afe7a0edb90b465eabc8111fef1c0d5650a03f6c0bc1b" Oct 30 21:33:27.817336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.817319 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" Oct 30 21:33:27.817441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.817361 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt" event={"ID":"f36a4d63-9c35-4824-9cf1-77c96d325cac","Type":"ContainerDied","Data":"af11d1178ad446625eb6f8e4e523dac1075121b1f69852fe20a21e6d4a12b148"} Oct 30 21:33:27.817441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:27.817396 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af11d1178ad446625eb6f8e4e523dac1075121b1f69852fe20a21e6d4a12b148" Oct 30 21:33:30.771613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.771575 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww"] Oct 30 21:33:30.772072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772059 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a9c5f415-2ae4-419f-8b5d-9147a4a41204" containerName="revoke-keys" Oct 30 21:33:30.772072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772073 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c5f415-2ae4-419f-8b5d-9147a4a41204" containerName="revoke-keys" Oct 30 21:33:30.772173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772091 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f36a4d63-9c35-4824-9cf1-77c96d325cac" containerName="revoke-keys" Oct 30 21:33:30.772173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772096 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36a4d63-9c35-4824-9cf1-77c96d325cac" containerName="revoke-keys" Oct 30 21:33:30.772173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772115 4526 cpu_manager.go:414] "containerMap: removing container" podUID="71dd9c35-5a0d-42ea-a30f-633a2c95fe3c" containerName="revoke-keys" Oct 30 21:33:30.772173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772120 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71dd9c35-5a0d-42ea-a30f-633a2c95fe3c" containerName="revoke-keys" Oct 30 21:33:30.772302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772191 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c5f415-2ae4-419f-8b5d-9147a4a41204" containerName="revoke-keys" Oct 30 21:33:30.772302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772202 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36a4d63-9c35-4824-9cf1-77c96d325cac" containerName="revoke-keys" Oct 30 21:33:30.772302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.772211 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="71dd9c35-5a0d-42ea-a30f-633a2c95fe3c" containerName="revoke-keys" Oct 30 21:33:30.776520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.776478 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:30.854248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.854216 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:30.854248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.854251 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:30.854471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.854360 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9wfh\" (UniqueName: \"kubernetes.io/projected/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-kube-api-access-z9wfh\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:30.856714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.856684 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:33:30.859445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.859423 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww"] Oct 30 21:33:30.878654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.878609 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:33:30.878654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.878625 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:33:30.878654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.878616 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:30.955785 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.955754 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:30.955985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.955795 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:30.955985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.955864 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9wfh\" (UniqueName: \"kubernetes.io/projected/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-kube-api-access-z9wfh\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:30.956518 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.956477 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:30.958292 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:30.958274 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:31.012189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:31.012153 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9wfh\" (UniqueName: \"kubernetes.io/projected/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-kube-api-access-z9wfh\") pod \"maas-api-revoke-sub-80a4d1fe-cj4ww\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:31.089351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:31.089325 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:31.462803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:31.462777 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww"] Oct 30 21:33:31.464762 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:31.464734 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2aa8c58_98c7_41ba_b7f2_0b0842dbf0bd.slice/crio-6f8fcd8360d532c9ef2a806c4cd487986ecc187bbb51e9f4b568547ff2097b08 WatchSource:0}: Error finding container 6f8fcd8360d532c9ef2a806c4cd487986ecc187bbb51e9f4b568547ff2097b08: Status 404 returned error can't find the container with id 6f8fcd8360d532c9ef2a806c4cd487986ecc187bbb51e9f4b568547ff2097b08 Oct 30 21:33:31.845212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:31.845164 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" event={"ID":"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd","Type":"ContainerStarted","Data":"8e1e8a09eab3b1e87d5879f8104335b11289ebf85feecc39302ee6d17181c862"} Oct 30 21:33:31.845680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:31.845235 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" event={"ID":"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd","Type":"ContainerStarted","Data":"6f8fcd8360d532c9ef2a806c4cd487986ecc187bbb51e9f4b568547ff2097b08"} Oct 30 21:33:31.960041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:31.959985 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" podStartSLOduration=1.959969699 podStartE2EDuration="1.959969699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:31.958745031 +0000 UTC m=+1002.092118229" watchObservedRunningTime="2026-10-30 21:33:31.959969699 +0000 UTC m=+1002.093342897" Oct 30 21:33:32.853233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:32.853194 4526 generic.go:356] "Generic (PLEG): container finished" podID="a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd" containerID="8e1e8a09eab3b1e87d5879f8104335b11289ebf85feecc39302ee6d17181c862" exitCode=0 Oct 30 21:33:32.853696 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:32.853273 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" event={"ID":"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd","Type":"ContainerDied","Data":"8e1e8a09eab3b1e87d5879f8104335b11289ebf85feecc39302ee6d17181c862"} Oct 30 21:33:34.008101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.008072 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:34.090300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.090264 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-xks-ca\") pod \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " Oct 30 21:33:34.090535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.090357 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-openshift-ca\") pod \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " Oct 30 21:33:34.090535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.090432 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-kube-api-access-z9wfh\" (UniqueName: \"kubernetes.io/projected/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-kube-api-access-z9wfh\") pod \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\" (UID: \"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd\") " Oct 30 21:33:34.090776 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.090747 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-openshift-ca" pod "a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd" (UID: "a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:34.090978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.090956 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:34.092793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.092768 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-xks-ca" pod "a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd" (UID: "a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:34.092862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.092809 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-kube-api-access-z9wfh" pod "a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd" (UID: "a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd"). InnerVolumeSpecName "kube-api-access-z9wfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:34.192204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.192117 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:34.192204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.192146 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9wfh\" (UniqueName: \"kubernetes.io/projected/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd-kube-api-access-z9wfh\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:34.866694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.866660 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" Oct 30 21:33:34.866694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.866671 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww" event={"ID":"a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd","Type":"ContainerDied","Data":"6f8fcd8360d532c9ef2a806c4cd487986ecc187bbb51e9f4b568547ff2097b08"} Oct 30 21:33:34.866922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:34.866724 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f8fcd8360d532c9ef2a806c4cd487986ecc187bbb51e9f4b568547ff2097b08" Oct 30 21:33:35.581926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.581896 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w"] Oct 30 21:33:35.582540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.582522 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd" containerName="revoke-keys" Oct 30 21:33:35.582626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.582541 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd" containerName="revoke-keys" Oct 30 21:33:35.582688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.582626 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd" containerName="revoke-keys" Oct 30 21:33:35.585947 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.585931 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.641615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.641583 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:33:35.642019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.641999 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:35.693786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.693755 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w"] Oct 30 21:33:35.708764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.708724 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkg8w\" (UniqueName: \"kubernetes.io/projected/e8900c9c-b546-47e9-afe9-ad4fec778605-kube-api-access-zkg8w\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.708764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.708771 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.709036 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.708943 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.741393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.741364 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:33:35.741540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.741447 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:33:35.809790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.809745 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.809994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.809895 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkg8w\" (UniqueName: \"kubernetes.io/projected/e8900c9c-b546-47e9-afe9-ad4fec778605-kube-api-access-zkg8w\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.809994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.809923 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.810421 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.810401 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.813225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.813193 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.867130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.867062 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkg8w\" (UniqueName: \"kubernetes.io/projected/e8900c9c-b546-47e9-afe9-ad4fec778605-kube-api-access-zkg8w\") pod \"maas-api-revoke-sub-4938380d-hnf4w\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:35.904239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:35.904201 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:36.282082 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:36.282058 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w"] Oct 30 21:33:36.283817 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:36.283789 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8900c9c_b546_47e9_afe9_ad4fec778605.slice/crio-0d75c36926076ede42c0c415fe2c53f18a67aa6a442f8076e8c20a7f12716f2a WatchSource:0}: Error finding container 0d75c36926076ede42c0c415fe2c53f18a67aa6a442f8076e8c20a7f12716f2a: Status 404 returned error can't find the container with id 0d75c36926076ede42c0c415fe2c53f18a67aa6a442f8076e8c20a7f12716f2a Oct 30 21:33:36.880556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:36.880432 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" event={"ID":"e8900c9c-b546-47e9-afe9-ad4fec778605","Type":"ContainerStarted","Data":"473f9d5b7188a77d1b8126f404a59f3293e7aeff6b1d906ef9f7247bff824de1"} Oct 30 21:33:36.880556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:36.880481 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" event={"ID":"e8900c9c-b546-47e9-afe9-ad4fec778605","Type":"ContainerStarted","Data":"0d75c36926076ede42c0c415fe2c53f18a67aa6a442f8076e8c20a7f12716f2a"} Oct 30 21:33:36.993460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:36.993304 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" podStartSLOduration=1.993284327 podStartE2EDuration="1.993284327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:36.992595763 +0000 UTC m=+1007.125968954" watchObservedRunningTime="2026-10-30 21:33:36.993284327 +0000 UTC m=+1007.126657526" Oct 30 21:33:37.887380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:37.887347 4526 generic.go:356] "Generic (PLEG): container finished" podID="e8900c9c-b546-47e9-afe9-ad4fec778605" containerID="473f9d5b7188a77d1b8126f404a59f3293e7aeff6b1d906ef9f7247bff824de1" exitCode=0 Oct 30 21:33:37.887812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:37.887427 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" event={"ID":"e8900c9c-b546-47e9-afe9-ad4fec778605","Type":"ContainerDied","Data":"473f9d5b7188a77d1b8126f404a59f3293e7aeff6b1d906ef9f7247bff824de1"} Oct 30 21:33:39.039558 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.039531 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:39.146770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.146688 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8900c9c-b546-47e9-afe9-ad4fec778605-kube-api-access-zkg8w\" (UniqueName: \"kubernetes.io/projected/e8900c9c-b546-47e9-afe9-ad4fec778605-kube-api-access-zkg8w\") pod \"e8900c9c-b546-47e9-afe9-ad4fec778605\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " Oct 30 21:33:39.146934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.146777 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-xks-ca\") pod \"e8900c9c-b546-47e9-afe9-ad4fec778605\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " Oct 30 21:33:39.146934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.146894 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-openshift-ca\") pod \"e8900c9c-b546-47e9-afe9-ad4fec778605\" (UID: \"e8900c9c-b546-47e9-afe9-ad4fec778605\") " Oct 30 21:33:39.147203 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.147177 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-openshift-ca" pod "e8900c9c-b546-47e9-afe9-ad4fec778605" (UID: "e8900c9c-b546-47e9-afe9-ad4fec778605"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:39.148916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.148897 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8900c9c-b546-47e9-afe9-ad4fec778605-kube-api-access-zkg8w" pod "e8900c9c-b546-47e9-afe9-ad4fec778605" (UID: "e8900c9c-b546-47e9-afe9-ad4fec778605"). InnerVolumeSpecName "kube-api-access-zkg8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:39.148961 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.148910 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-xks-ca" pod "e8900c9c-b546-47e9-afe9-ad4fec778605" (UID: "e8900c9c-b546-47e9-afe9-ad4fec778605"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:39.248147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.248115 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:39.248147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.248142 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkg8w\" (UniqueName: \"kubernetes.io/projected/e8900c9c-b546-47e9-afe9-ad4fec778605-kube-api-access-zkg8w\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:39.248147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.248153 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8900c9c-b546-47e9-afe9-ad4fec778605-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:39.899573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.899544 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" Oct 30 21:33:39.899739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.899589 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w" event={"ID":"e8900c9c-b546-47e9-afe9-ad4fec778605","Type":"ContainerDied","Data":"0d75c36926076ede42c0c415fe2c53f18a67aa6a442f8076e8c20a7f12716f2a"} Oct 30 21:33:39.899739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:39.899663 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d75c36926076ede42c0c415fe2c53f18a67aa6a442f8076e8c20a7f12716f2a" Oct 30 21:33:40.968437 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:40.968402 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq"] Oct 30 21:33:40.968982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:40.968967 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e8900c9c-b546-47e9-afe9-ad4fec778605" containerName="revoke-keys" Oct 30 21:33:40.969023 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:40.968983 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8900c9c-b546-47e9-afe9-ad4fec778605" containerName="revoke-keys" Oct 30 21:33:40.969078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:40.969068 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8900c9c-b546-47e9-afe9-ad4fec778605" containerName="revoke-keys" Oct 30 21:33:40.972849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:40.972828 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.026432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.026400 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:33:41.026432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.026400 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:41.026432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.026401 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:33:41.026756 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.026459 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:33:41.067934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.067900 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8vl\" (UniqueName: \"kubernetes.io/projected/a362fbd9-09f8-4bd3-8f87-36af30a08b33-kube-api-access-2l8vl\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.068104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.068048 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.068209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.068191 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.079392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.079362 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq"] Oct 30 21:33:41.169570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.169534 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.169738 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.169652 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.169738 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.169694 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8vl\" (UniqueName: \"kubernetes.io/projected/a362fbd9-09f8-4bd3-8f87-36af30a08b33-kube-api-access-2l8vl\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.170282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.170256 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.172089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.172068 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.226340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.226256 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8vl\" (UniqueName: \"kubernetes.io/projected/a362fbd9-09f8-4bd3-8f87-36af30a08b33-kube-api-access-2l8vl\") pod \"maas-api-revoke-sub-49bee2d1-stpjq\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.287990 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.287953 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:41.670674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.670653 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq"] Oct 30 21:33:41.673705 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:41.673672 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda362fbd9_09f8_4bd3_8f87_36af30a08b33.slice/crio-b984f00bfeb56e72db04c1170ef66a54144be396578a49108b086b41d5c92f07 WatchSource:0}: Error finding container b984f00bfeb56e72db04c1170ef66a54144be396578a49108b086b41d5c92f07: Status 404 returned error can't find the container with id b984f00bfeb56e72db04c1170ef66a54144be396578a49108b086b41d5c92f07 Oct 30 21:33:41.914915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:41.914883 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" event={"ID":"a362fbd9-09f8-4bd3-8f87-36af30a08b33","Type":"ContainerStarted","Data":"b984f00bfeb56e72db04c1170ef66a54144be396578a49108b086b41d5c92f07"} Oct 30 21:33:42.476944 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:33:42.476909 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:33:42.921770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:42.921733 4526 generic.go:356] "Generic (PLEG): container finished" podID="a362fbd9-09f8-4bd3-8f87-36af30a08b33" containerID="5a9e03f54ccee88b21e4f064331638ca64f83a66fa2aaf0ea9e8ea3ab476cf2d" exitCode=0 Oct 30 21:33:42.921946 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:42.921821 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" event={"ID":"a362fbd9-09f8-4bd3-8f87-36af30a08b33","Type":"ContainerDied","Data":"5a9e03f54ccee88b21e4f064331638ca64f83a66fa2aaf0ea9e8ea3ab476cf2d"} Oct 30 21:33:43.025552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.025512 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk"] Oct 30 21:33:43.029333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.029305 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.074069 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.074034 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk"] Oct 30 21:33:43.192698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.192603 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzs7\" (UniqueName: \"kubernetes.io/projected/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-kube-api-access-jmzs7\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.192873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.192721 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.192873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.192795 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.293668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.293628 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.293849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.293718 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzs7\" (UniqueName: \"kubernetes.io/projected/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-kube-api-access-jmzs7\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.293849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.293761 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.294364 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.294345 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.296228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.296208 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.350469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.350437 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzs7\" (UniqueName: \"kubernetes.io/projected/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-kube-api-access-jmzs7\") pod \"maas-api-revoke-sub-41d466b5-jr4lk\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:43.645656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:43.645616 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:44.026005 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.025971 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk"] Oct 30 21:33:44.027533 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:44.027503 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dc0e6c_dad3_4b28_b2fa_89b60c5a3208.slice/crio-268684f1cc1ad1bf9c08f5d483a929f5d5328bd363a56080d1108881c56da60e WatchSource:0}: Error finding container 268684f1cc1ad1bf9c08f5d483a929f5d5328bd363a56080d1108881c56da60e: Status 404 returned error can't find the container with id 268684f1cc1ad1bf9c08f5d483a929f5d5328bd363a56080d1108881c56da60e Oct 30 21:33:44.063481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.063459 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:44.203070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.202987 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-xks-ca\") pod \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " Oct 30 21:33:44.203070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.203035 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a362fbd9-09f8-4bd3-8f87-36af30a08b33-kube-api-access-2l8vl\" (UniqueName: \"kubernetes.io/projected/a362fbd9-09f8-4bd3-8f87-36af30a08b33-kube-api-access-2l8vl\") pod \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " Oct 30 21:33:44.203287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.203170 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-openshift-ca\") pod \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\" (UID: \"a362fbd9-09f8-4bd3-8f87-36af30a08b33\") " Oct 30 21:33:44.203790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.203761 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-openshift-ca" pod "a362fbd9-09f8-4bd3-8f87-36af30a08b33" (UID: "a362fbd9-09f8-4bd3-8f87-36af30a08b33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:44.206260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.205786 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-xks-ca" pod "a362fbd9-09f8-4bd3-8f87-36af30a08b33" (UID: "a362fbd9-09f8-4bd3-8f87-36af30a08b33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:44.206260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.205893 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a362fbd9-09f8-4bd3-8f87-36af30a08b33-kube-api-access-2l8vl" pod "a362fbd9-09f8-4bd3-8f87-36af30a08b33" (UID: "a362fbd9-09f8-4bd3-8f87-36af30a08b33"). InnerVolumeSpecName "kube-api-access-2l8vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:44.304745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.304551 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:44.304745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.304579 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l8vl\" (UniqueName: \"kubernetes.io/projected/a362fbd9-09f8-4bd3-8f87-36af30a08b33-kube-api-access-2l8vl\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:44.304745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.304596 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a362fbd9-09f8-4bd3-8f87-36af30a08b33-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:44.871536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.871483 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7"] Oct 30 21:33:44.871977 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.871965 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a362fbd9-09f8-4bd3-8f87-36af30a08b33" containerName="revoke-keys" Oct 30 21:33:44.872018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.871978 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a362fbd9-09f8-4bd3-8f87-36af30a08b33" containerName="revoke-keys" Oct 30 21:33:44.872083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.872074 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a362fbd9-09f8-4bd3-8f87-36af30a08b33" containerName="revoke-keys" Oct 30 21:33:44.875176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.875162 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:44.935574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.935550 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" Oct 30 21:33:44.935739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.935591 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq" event={"ID":"a362fbd9-09f8-4bd3-8f87-36af30a08b33","Type":"ContainerDied","Data":"b984f00bfeb56e72db04c1170ef66a54144be396578a49108b086b41d5c92f07"} Oct 30 21:33:44.935739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.935638 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b984f00bfeb56e72db04c1170ef66a54144be396578a49108b086b41d5c92f07" Oct 30 21:33:44.937570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.937541 4526 generic.go:356] "Generic (PLEG): container finished" podID="76dc0e6c-dad3-4b28-b2fa-89b60c5a3208" containerID="6feb599eecb45174152529c401df3f09cb38ba18b4caba9397b80fe7500787c8" exitCode=0 Oct 30 21:33:44.937721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.937598 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" event={"ID":"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208","Type":"ContainerDied","Data":"6feb599eecb45174152529c401df3f09cb38ba18b4caba9397b80fe7500787c8"} Oct 30 21:33:44.937721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.937640 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" event={"ID":"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208","Type":"ContainerStarted","Data":"268684f1cc1ad1bf9c08f5d483a929f5d5328bd363a56080d1108881c56da60e"} Oct 30 21:33:44.978202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.978157 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"default-dockercfg-s2ptq\"" Oct 30 21:33:44.978202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.978188 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"kube-root-ca.crt\"" Oct 30 21:33:44.978442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.978236 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"distinct-w4-07fa8e-kserve-self-signed-certs\"" Oct 30 21:33:44.978442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.978284 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"openshift-service-ca.crt\"" Oct 30 21:33:44.981377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:44.981359 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7"] Oct 30 21:33:45.012222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.012188 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tmp-dir\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.012407 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.012236 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-model-cache\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.012407 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.012253 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tls-certs\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.012482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.012409 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-home\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.012482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.012451 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-dshm\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.012553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.012504 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5rt\" (UniqueName: \"kubernetes.io/projected/a83fd981-a59c-4a55-b675-d19b4bab1d5e-kube-api-access-9j5rt\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113276 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tmp-dir\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113331 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-model-cache\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113354 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tls-certs\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113401 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-home\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113429 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-dshm\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113507 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5rt\" (UniqueName: \"kubernetes.io/projected/a83fd981-a59c-4a55-b675-d19b4bab1d5e-kube-api-access-9j5rt\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113790 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-model-cache\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113812 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tmp-dir\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.113909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.113882 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-home\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.115942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.115926 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-dshm\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.116063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.116045 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tls-certs\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.171779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.171748 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5rt\" (UniqueName: \"kubernetes.io/projected/a83fd981-a59c-4a55-b675-d19b4bab1d5e-kube-api-access-9j5rt\") pod \"distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.234985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.234952 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:45.625963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.625939 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7"] Oct 30 21:33:45.627311 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:45.627278 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda83fd981_a59c_4a55_b675_d19b4bab1d5e.slice/crio-092aba4c87a0e3fec57d4a3a60eda8ef4be93fcc329b1b2c8a0f2ff7114c7063 WatchSource:0}: Error finding container 092aba4c87a0e3fec57d4a3a60eda8ef4be93fcc329b1b2c8a0f2ff7114c7063: Status 404 returned error can't find the container with id 092aba4c87a0e3fec57d4a3a60eda8ef4be93fcc329b1b2c8a0f2ff7114c7063 Oct 30 21:33:45.945929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.945829 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" event={"ID":"a83fd981-a59c-4a55-b675-d19b4bab1d5e","Type":"ContainerStarted","Data":"d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50"} Oct 30 21:33:45.945929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:45.945879 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" event={"ID":"a83fd981-a59c-4a55-b675-d19b4bab1d5e","Type":"ContainerStarted","Data":"092aba4c87a0e3fec57d4a3a60eda8ef4be93fcc329b1b2c8a0f2ff7114c7063"} Oct 30 21:33:46.060704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.060650 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podStartSLOduration=2.060625576 podStartE2EDuration="2.060625576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:46.060207282 +0000 UTC m=+1016.193580480" watchObservedRunningTime="2026-10-30 21:33:46.060625576 +0000 UTC m=+1016.193998808" Oct 30 21:33:46.102837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.102813 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:46.223505 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.223405 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-kube-api-access-jmzs7\" (UniqueName: \"kubernetes.io/projected/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-kube-api-access-jmzs7\") pod \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " Oct 30 21:33:46.223676 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.223514 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-openshift-ca\") pod \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " Oct 30 21:33:46.223676 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.223564 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-xks-ca\") pod \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\" (UID: \"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208\") " Oct 30 21:33:46.225907 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.223903 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-openshift-ca" pod "76dc0e6c-dad3-4b28-b2fa-89b60c5a3208" (UID: "76dc0e6c-dad3-4b28-b2fa-89b60c5a3208"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:46.225907 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.224052 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:46.226416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.226389 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-kube-api-access-jmzs7" pod "76dc0e6c-dad3-4b28-b2fa-89b60c5a3208" (UID: "76dc0e6c-dad3-4b28-b2fa-89b60c5a3208"). InnerVolumeSpecName "kube-api-access-jmzs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:46.226537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.226453 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-xks-ca" pod "76dc0e6c-dad3-4b28-b2fa-89b60c5a3208" (UID: "76dc0e6c-dad3-4b28-b2fa-89b60c5a3208"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:46.235038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.235017 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:46.325387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.325357 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmzs7\" (UniqueName: \"kubernetes.io/projected/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-kube-api-access-jmzs7\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:46.325387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.325386 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:46.952701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.952672 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" Oct 30 21:33:46.952701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.952694 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk" event={"ID":"76dc0e6c-dad3-4b28-b2fa-89b60c5a3208","Type":"ContainerDied","Data":"268684f1cc1ad1bf9c08f5d483a929f5d5328bd363a56080d1108881c56da60e"} Oct 30 21:33:46.953224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:46.952738 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="268684f1cc1ad1bf9c08f5d483a929f5d5328bd363a56080d1108881c56da60e" Oct 30 21:33:52.722885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.722848 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk"] Oct 30 21:33:52.723356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.723342 4526 cpu_manager.go:414] "containerMap: removing container" podUID="76dc0e6c-dad3-4b28-b2fa-89b60c5a3208" containerName="revoke-keys" Oct 30 21:33:52.723399 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.723357 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76dc0e6c-dad3-4b28-b2fa-89b60c5a3208" containerName="revoke-keys" Oct 30 21:33:52.723466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.723457 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="76dc0e6c-dad3-4b28-b2fa-89b60c5a3208" containerName="revoke-keys" Oct 30 21:33:52.727067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.727051 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:52.783411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.783376 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk"] Oct 30 21:33:52.828812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.828769 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:52.828812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.828775 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:33:52.829028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.828848 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:33:52.829028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.828956 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:33:52.898754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.898721 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:52.898923 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.898771 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:52.898969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.898924 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgxf\" (UniqueName: \"kubernetes.io/projected/3996407b-fd70-401a-a329-ad9ab6542ef9-kube-api-access-7pgxf\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:52.999697 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.999616 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:52.999848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.999717 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgxf\" (UniqueName: \"kubernetes.io/projected/3996407b-fd70-401a-a329-ad9ab6542ef9-kube-api-access-7pgxf\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:52.999848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:52.999782 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:53.000341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:53.000310 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:53.002283 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:53.002267 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:53.055775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:53.055748 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgxf\" (UniqueName: \"kubernetes.io/projected/3996407b-fd70-401a-a329-ad9ab6542ef9-kube-api-access-7pgxf\") pod \"maas-api-revoke-sub-70412924-cgxgk\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:53.339895 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:53.339864 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:53.721614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:53.721590 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk"] Oct 30 21:33:53.723927 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:53.723899 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3996407b_fd70_401a_a329_ad9ab6542ef9.slice/crio-d0a0772e4b3ca8f3c85c40e158930c0b86c38323043d0612d2e46a4e7977206a WatchSource:0}: Error finding container d0a0772e4b3ca8f3c85c40e158930c0b86c38323043d0612d2e46a4e7977206a: Status 404 returned error can't find the container with id d0a0772e4b3ca8f3c85c40e158930c0b86c38323043d0612d2e46a4e7977206a Oct 30 21:33:53.996554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:53.996519 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" event={"ID":"3996407b-fd70-401a-a329-ad9ab6542ef9","Type":"ContainerStarted","Data":"d0a0772e4b3ca8f3c85c40e158930c0b86c38323043d0612d2e46a4e7977206a"} Oct 30 21:33:55.004642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:55.004540 4526 generic.go:356] "Generic (PLEG): container finished" podID="3996407b-fd70-401a-a329-ad9ab6542ef9" containerID="71ed0323d530f9a0ac6c21bff5e1b1fa74f9b2b7576793626d5a8435e64f73e8" exitCode=0 Oct 30 21:33:55.005024 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:55.004621 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" event={"ID":"3996407b-fd70-401a-a329-ad9ab6542ef9","Type":"ContainerDied","Data":"71ed0323d530f9a0ac6c21bff5e1b1fa74f9b2b7576793626d5a8435e64f73e8"} Oct 30 21:33:55.235541 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:55.235471 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:55.248359 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:55.248332 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:56.022950 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.022919 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:33:56.161929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.161903 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:56.235563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.235533 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-xks-ca\") pod \"3996407b-fd70-401a-a329-ad9ab6542ef9\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " Oct 30 21:33:56.235761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.235614 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-openshift-ca\") pod \"3996407b-fd70-401a-a329-ad9ab6542ef9\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " Oct 30 21:33:56.235761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.235707 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3996407b-fd70-401a-a329-ad9ab6542ef9-kube-api-access-7pgxf\" (UniqueName: \"kubernetes.io/projected/3996407b-fd70-401a-a329-ad9ab6542ef9-kube-api-access-7pgxf\") pod \"3996407b-fd70-401a-a329-ad9ab6542ef9\" (UID: \"3996407b-fd70-401a-a329-ad9ab6542ef9\") " Oct 30 21:33:56.235985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.235961 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-openshift-ca" pod "3996407b-fd70-401a-a329-ad9ab6542ef9" (UID: "3996407b-fd70-401a-a329-ad9ab6542ef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:56.237831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.237803 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-xks-ca" pod "3996407b-fd70-401a-a329-ad9ab6542ef9" (UID: "3996407b-fd70-401a-a329-ad9ab6542ef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:56.237944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.237912 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3996407b-fd70-401a-a329-ad9ab6542ef9-kube-api-access-7pgxf" pod "3996407b-fd70-401a-a329-ad9ab6542ef9" (UID: "3996407b-fd70-401a-a329-ad9ab6542ef9"). InnerVolumeSpecName "kube-api-access-7pgxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:56.336763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.336725 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:56.336763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.336756 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pgxf\" (UniqueName: \"kubernetes.io/projected/3996407b-fd70-401a-a329-ad9ab6542ef9-kube-api-access-7pgxf\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:56.336763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:56.336767 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3996407b-fd70-401a-a329-ad9ab6542ef9-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:33:57.020426 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.020394 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" event={"ID":"3996407b-fd70-401a-a329-ad9ab6542ef9","Type":"ContainerDied","Data":"d0a0772e4b3ca8f3c85c40e158930c0b86c38323043d0612d2e46a4e7977206a"} Oct 30 21:33:57.020650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.020440 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0a0772e4b3ca8f3c85c40e158930c0b86c38323043d0612d2e46a4e7977206a" Oct 30 21:33:57.022276 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.022251 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8"] Oct 30 21:33:57.022984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.022968 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3996407b-fd70-401a-a329-ad9ab6542ef9" containerName="revoke-keys" Oct 30 21:33:57.023391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.022986 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3996407b-fd70-401a-a329-ad9ab6542ef9" containerName="revoke-keys" Oct 30 21:33:57.023391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.023148 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3996407b-fd70-401a-a329-ad9ab6542ef9" containerName="revoke-keys" Oct 30 21:33:57.023391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.023342 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk" Oct 30 21:33:57.027068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.027053 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.045058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.045028 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/9ec58bb5-7c6f-4190-936c-57f6b01720dd-kube-api-access-pctff\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.045229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.045172 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.045297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.045266 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.081973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.081943 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8"] Oct 30 21:33:57.126958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.126928 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:57.126958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.126936 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:33:57.127179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.126937 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:33:57.128387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.128370 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:33:57.146047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.146019 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/9ec58bb5-7c6f-4190-936c-57f6b01720dd-kube-api-access-pctff\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.146188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.146092 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.146188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.146123 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.146710 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.146692 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.148650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.148620 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.205427 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.205394 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/9ec58bb5-7c6f-4190-936c-57f6b01720dd-kube-api-access-pctff\") pod \"maas-api-revoke-sub-9f26399c-bpnk8\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.342150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.342099 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:33:57.718480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:57.718450 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8"] Oct 30 21:33:57.719762 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:57.719736 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec58bb5_7c6f_4190_936c_57f6b01720dd.slice/crio-96064b4dc1cc423a5086112c772b0bb951863d0463b44bdfdaa0e8860a041315 WatchSource:0}: Error finding container 96064b4dc1cc423a5086112c772b0bb951863d0463b44bdfdaa0e8860a041315: Status 404 returned error can't find the container with id 96064b4dc1cc423a5086112c772b0bb951863d0463b44bdfdaa0e8860a041315 Oct 30 21:33:58.027790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.027756 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" event={"ID":"9ec58bb5-7c6f-4190-936c-57f6b01720dd","Type":"ContainerStarted","Data":"d2ff5addbb7d5a6885820eb60fdcdea72bfa3b312131d47ce967a1ff6302ef0f"} Oct 30 21:33:58.028174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.027811 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" event={"ID":"9ec58bb5-7c6f-4190-936c-57f6b01720dd","Type":"ContainerStarted","Data":"96064b4dc1cc423a5086112c772b0bb951863d0463b44bdfdaa0e8860a041315"} Oct 30 21:33:58.140900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.140841 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" podStartSLOduration=2.140821458 podStartE2EDuration="2.140821458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:58.139832142 +0000 UTC m=+1028.273205340" watchObservedRunningTime="2026-10-30 21:33:58.140821458 +0000 UTC m=+1028.274194658" Oct 30 21:33:58.749211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.749182 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc"] Oct 30 21:33:58.753009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.752993 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.765023 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.765000 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4r4w\" (UniqueName: \"kubernetes.io/projected/45438f16-6480-4add-aa65-5f5e8a6865bb-kube-api-access-f4r4w\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.765137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.765044 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.765137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.765132 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.810658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.810625 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc"] Oct 30 21:33:58.866677 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.866638 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4r4w\" (UniqueName: \"kubernetes.io/projected/45438f16-6480-4add-aa65-5f5e8a6865bb-kube-api-access-f4r4w\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.866851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.866690 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.866851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.866739 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.867341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.867317 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.869359 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.869330 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:58.923386 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:58.923311 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4r4w\" (UniqueName: \"kubernetes.io/projected/45438f16-6480-4add-aa65-5f5e8a6865bb-kube-api-access-f4r4w\") pod \"maas-api-revoke-sub-c216b558-8nvwc\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:59.036094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:59.036064 4526 generic.go:356] "Generic (PLEG): container finished" podID="9ec58bb5-7c6f-4190-936c-57f6b01720dd" containerID="d2ff5addbb7d5a6885820eb60fdcdea72bfa3b312131d47ce967a1ff6302ef0f" exitCode=0 Oct 30 21:33:59.036503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:59.036135 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" event={"ID":"9ec58bb5-7c6f-4190-936c-57f6b01720dd","Type":"ContainerDied","Data":"d2ff5addbb7d5a6885820eb60fdcdea72bfa3b312131d47ce967a1ff6302ef0f"} Oct 30 21:33:59.065620 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:59.065591 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:33:59.440768 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:33:59.440742 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc"] Oct 30 21:33:59.442233 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:33:59.442205 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45438f16_6480_4add_aa65_5f5e8a6865bb.slice/crio-9bb33699fb39171fd76b24689b19eb62b921f2e5a9ebac8c33c4ddd208959ae7 WatchSource:0}: Error finding container 9bb33699fb39171fd76b24689b19eb62b921f2e5a9ebac8c33c4ddd208959ae7: Status 404 returned error can't find the container with id 9bb33699fb39171fd76b24689b19eb62b921f2e5a9ebac8c33c4ddd208959ae7 Oct 30 21:34:00.043618 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.043561 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" event={"ID":"45438f16-6480-4add-aa65-5f5e8a6865bb","Type":"ContainerStarted","Data":"6208a37b3ca74027b37972badf099f7d960d0ce19039d982bf8d6b10440bdabe"} Oct 30 21:34:00.044062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.043638 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" event={"ID":"45438f16-6480-4add-aa65-5f5e8a6865bb","Type":"ContainerStarted","Data":"9bb33699fb39171fd76b24689b19eb62b921f2e5a9ebac8c33c4ddd208959ae7"} Oct 30 21:34:00.160245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.160160 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" podStartSLOduration=2.160141303 podStartE2EDuration="2.160141303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:00.157833222 +0000 UTC m=+1030.291206419" watchObservedRunningTime="2026-10-30 21:34:00.160141303 +0000 UTC m=+1030.293514512" Oct 30 21:34:00.197461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.197436 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:34:00.285158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.285127 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec58bb5-7c6f-4190-936c-57f6b01720dd-kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/9ec58bb5-7c6f-4190-936c-57f6b01720dd-kube-api-access-pctff\") pod \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " Oct 30 21:34:00.285351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.285202 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-xks-ca\") pod \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " Oct 30 21:34:00.285419 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.285402 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-openshift-ca\") pod \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\" (UID: \"9ec58bb5-7c6f-4190-936c-57f6b01720dd\") " Oct 30 21:34:00.285792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.285754 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-openshift-ca" pod "9ec58bb5-7c6f-4190-936c-57f6b01720dd" (UID: "9ec58bb5-7c6f-4190-936c-57f6b01720dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:00.285933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.285864 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:00.287546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.287523 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-xks-ca" pod "9ec58bb5-7c6f-4190-936c-57f6b01720dd" (UID: "9ec58bb5-7c6f-4190-936c-57f6b01720dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:00.287636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.287563 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec58bb5-7c6f-4190-936c-57f6b01720dd-kube-api-access-pctff" pod "9ec58bb5-7c6f-4190-936c-57f6b01720dd" (UID: "9ec58bb5-7c6f-4190-936c-57f6b01720dd"). InnerVolumeSpecName "kube-api-access-pctff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:00.387393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.387362 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/9ec58bb5-7c6f-4190-936c-57f6b01720dd-kube-api-access-pctff\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:00.387393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:00.387391 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec58bb5-7c6f-4190-936c-57f6b01720dd-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:01.050759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:01.050729 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" Oct 30 21:34:01.050759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:01.050740 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8" event={"ID":"9ec58bb5-7c6f-4190-936c-57f6b01720dd","Type":"ContainerDied","Data":"96064b4dc1cc423a5086112c772b0bb951863d0463b44bdfdaa0e8860a041315"} Oct 30 21:34:01.051273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:01.050797 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96064b4dc1cc423a5086112c772b0bb951863d0463b44bdfdaa0e8860a041315" Oct 30 21:34:01.052410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:01.052379 4526 generic.go:356] "Generic (PLEG): container finished" podID="45438f16-6480-4add-aa65-5f5e8a6865bb" containerID="6208a37b3ca74027b37972badf099f7d960d0ce19039d982bf8d6b10440bdabe" exitCode=0 Oct 30 21:34:01.052668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:01.052422 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" event={"ID":"45438f16-6480-4add-aa65-5f5e8a6865bb","Type":"ContainerDied","Data":"6208a37b3ca74027b37972badf099f7d960d0ce19039d982bf8d6b10440bdabe"} Oct 30 21:34:02.206534 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.206479 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:34:02.307911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.307867 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-xks-ca\") pod \"45438f16-6480-4add-aa65-5f5e8a6865bb\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " Oct 30 21:34:02.308080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.307923 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-openshift-ca\") pod \"45438f16-6480-4add-aa65-5f5e8a6865bb\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " Oct 30 21:34:02.308080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.308061 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45438f16-6480-4add-aa65-5f5e8a6865bb-kube-api-access-f4r4w\" (UniqueName: \"kubernetes.io/projected/45438f16-6480-4add-aa65-5f5e8a6865bb-kube-api-access-f4r4w\") pod \"45438f16-6480-4add-aa65-5f5e8a6865bb\" (UID: \"45438f16-6480-4add-aa65-5f5e8a6865bb\") " Oct 30 21:34:02.308314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.308289 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-openshift-ca" pod "45438f16-6480-4add-aa65-5f5e8a6865bb" (UID: "45438f16-6480-4add-aa65-5f5e8a6865bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:02.308396 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.308357 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:02.310230 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.310170 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-xks-ca" pod "45438f16-6480-4add-aa65-5f5e8a6865bb" (UID: "45438f16-6480-4add-aa65-5f5e8a6865bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:02.310320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.310261 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45438f16-6480-4add-aa65-5f5e8a6865bb-kube-api-access-f4r4w" pod "45438f16-6480-4add-aa65-5f5e8a6865bb" (UID: "45438f16-6480-4add-aa65-5f5e8a6865bb"). InnerVolumeSpecName "kube-api-access-f4r4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:02.409040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.409005 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4r4w\" (UniqueName: \"kubernetes.io/projected/45438f16-6480-4add-aa65-5f5e8a6865bb-kube-api-access-f4r4w\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:02.409040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:02.409034 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45438f16-6480-4add-aa65-5f5e8a6865bb-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:03.069766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.069727 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" Oct 30 21:34:03.069766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.069742 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc" event={"ID":"45438f16-6480-4add-aa65-5f5e8a6865bb","Type":"ContainerDied","Data":"9bb33699fb39171fd76b24689b19eb62b921f2e5a9ebac8c33c4ddd208959ae7"} Oct 30 21:34:03.069982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.069799 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bb33699fb39171fd76b24689b19eb62b921f2e5a9ebac8c33c4ddd208959ae7" Oct 30 21:34:03.187059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.187027 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w"] Oct 30 21:34:03.187666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.187648 4526 cpu_manager.go:414] "containerMap: removing container" podUID="45438f16-6480-4add-aa65-5f5e8a6865bb" containerName="revoke-keys" Oct 30 21:34:03.187779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.187668 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45438f16-6480-4add-aa65-5f5e8a6865bb" containerName="revoke-keys" Oct 30 21:34:03.187779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.187700 4526 cpu_manager.go:414] "containerMap: removing container" podUID="9ec58bb5-7c6f-4190-936c-57f6b01720dd" containerName="revoke-keys" Oct 30 21:34:03.187779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.187709 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec58bb5-7c6f-4190-936c-57f6b01720dd" containerName="revoke-keys" Oct 30 21:34:03.187917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.187853 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="45438f16-6480-4add-aa65-5f5e8a6865bb" containerName="revoke-keys" Oct 30 21:34:03.187917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.187870 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec58bb5-7c6f-4190-936c-57f6b01720dd" containerName="revoke-keys" Oct 30 21:34:03.191339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.191317 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.219152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.218789 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.219152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.218875 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.219152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.218968 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5td9q\" (UniqueName: \"kubernetes.io/projected/bc159488-381b-4c46-8add-de64f8b8296a-kube-api-access-5td9q\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.266252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.266209 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:34:03.267513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.266950 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:34:03.267513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.267050 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:34:03.267513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.267145 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:34:03.271387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.271357 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w"] Oct 30 21:34:03.320010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.319906 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.320647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.320614 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.320898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.320876 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5td9q\" (UniqueName: \"kubernetes.io/projected/bc159488-381b-4c46-8add-de64f8b8296a-kube-api-access-5td9q\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.321673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.321653 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.323940 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.323912 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.378672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.378643 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5td9q\" (UniqueName: \"kubernetes.io/projected/bc159488-381b-4c46-8add-de64f8b8296a-kube-api-access-5td9q\") pod \"maas-api-revoke-sub-122eae23-mqg2w\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.524943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.524905 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:03.553513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.551844 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz"] Oct 30 21:34:03.579004 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.578921 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.613695 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.590264 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz"] Oct 30 21:34:03.668523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.652248 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.668523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.652319 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.668523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.652363 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27vt\" (UniqueName: \"kubernetes.io/projected/0e209c61-424e-4070-8358-6d4bc95ab185-kube-api-access-z27vt\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.756514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.756393 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.756514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.756456 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.756514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.756513 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27vt\" (UniqueName: \"kubernetes.io/projected/0e209c61-424e-4070-8358-6d4bc95ab185-kube-api-access-z27vt\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.758679 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.757319 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.766237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.766205 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.859583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.850233 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27vt\" (UniqueName: \"kubernetes.io/projected/0e209c61-424e-4070-8358-6d4bc95ab185-kube-api-access-z27vt\") pod \"maas-api-revoke-sub-a945c0fe-fccvz\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:03.974403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:03.974367 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:04.119625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:04.118758 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w"] Oct 30 21:34:04.394938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:04.394910 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz"] Oct 30 21:34:04.396215 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:34:04.396187 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e209c61_424e_4070_8358_6d4bc95ab185.slice/crio-80ccb72b56661f7c27ac0fcaf664ec9a89ed22f03b47efc73c1e81fe60079ccf WatchSource:0}: Error finding container 80ccb72b56661f7c27ac0fcaf664ec9a89ed22f03b47efc73c1e81fe60079ccf: Status 404 returned error can't find the container with id 80ccb72b56661f7c27ac0fcaf664ec9a89ed22f03b47efc73c1e81fe60079ccf Oct 30 21:34:05.087944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:05.087909 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" event={"ID":"0e209c61-424e-4070-8358-6d4bc95ab185","Type":"ContainerStarted","Data":"d892958e2f326deddf671917080fd9e6b2a77f4b0ee944ba5c07cc7c3d0b1ca9"} Oct 30 21:34:05.088155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:05.087964 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" event={"ID":"0e209c61-424e-4070-8358-6d4bc95ab185","Type":"ContainerStarted","Data":"80ccb72b56661f7c27ac0fcaf664ec9a89ed22f03b47efc73c1e81fe60079ccf"} Oct 30 21:34:05.089404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:05.089381 4526 generic.go:356] "Generic (PLEG): container finished" podID="bc159488-381b-4c46-8add-de64f8b8296a" containerID="4f1aa7682512dd32aa04bbb7f350b9073aad00198dd078f807d34e10d57dc338" exitCode=0 Oct 30 21:34:05.089508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:05.089459 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" event={"ID":"bc159488-381b-4c46-8add-de64f8b8296a","Type":"ContainerDied","Data":"4f1aa7682512dd32aa04bbb7f350b9073aad00198dd078f807d34e10d57dc338"} Oct 30 21:34:05.089553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:05.089526 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" event={"ID":"bc159488-381b-4c46-8add-de64f8b8296a","Type":"ContainerStarted","Data":"ff3cda574ffa06b569ec59c5c787faeebcae87accc871160372f65a18546b319"} Oct 30 21:34:05.200614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:05.200552 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" podStartSLOduration=2.200535094 podStartE2EDuration="2.200535094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:05.1997723 +0000 UTC m=+1035.333145500" watchObservedRunningTime="2026-10-30 21:34:05.200535094 +0000 UTC m=+1035.333908292" Oct 30 21:34:06.099028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.098995 4526 generic.go:356] "Generic (PLEG): container finished" podID="0e209c61-424e-4070-8358-6d4bc95ab185" containerID="d892958e2f326deddf671917080fd9e6b2a77f4b0ee944ba5c07cc7c3d0b1ca9" exitCode=0 Oct 30 21:34:06.099443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.099070 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" event={"ID":"0e209c61-424e-4070-8358-6d4bc95ab185","Type":"ContainerDied","Data":"d892958e2f326deddf671917080fd9e6b2a77f4b0ee944ba5c07cc7c3d0b1ca9"} Oct 30 21:34:06.238861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.238832 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:06.286325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.286299 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-openshift-ca\") pod \"bc159488-381b-4c46-8add-de64f8b8296a\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " Oct 30 21:34:06.286473 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.286369 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-xks-ca\") pod \"bc159488-381b-4c46-8add-de64f8b8296a\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " Oct 30 21:34:06.286473 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.286416 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc159488-381b-4c46-8add-de64f8b8296a-kube-api-access-5td9q\" (UniqueName: \"kubernetes.io/projected/bc159488-381b-4c46-8add-de64f8b8296a-kube-api-access-5td9q\") pod \"bc159488-381b-4c46-8add-de64f8b8296a\" (UID: \"bc159488-381b-4c46-8add-de64f8b8296a\") " Oct 30 21:34:06.286691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.286658 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-openshift-ca" pod "bc159488-381b-4c46-8add-de64f8b8296a" (UID: "bc159488-381b-4c46-8add-de64f8b8296a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:06.286878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.286864 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:06.288439 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.288412 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-xks-ca" pod "bc159488-381b-4c46-8add-de64f8b8296a" (UID: "bc159488-381b-4c46-8add-de64f8b8296a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:06.288577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.288473 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc159488-381b-4c46-8add-de64f8b8296a-kube-api-access-5td9q" pod "bc159488-381b-4c46-8add-de64f8b8296a" (UID: "bc159488-381b-4c46-8add-de64f8b8296a"). InnerVolumeSpecName "kube-api-access-5td9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:06.388348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.388281 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc159488-381b-4c46-8add-de64f8b8296a-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:06.388348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:06.388306 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5td9q\" (UniqueName: \"kubernetes.io/projected/bc159488-381b-4c46-8add-de64f8b8296a-kube-api-access-5td9q\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:07.106799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.106767 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" Oct 30 21:34:07.106799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.106766 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w" event={"ID":"bc159488-381b-4c46-8add-de64f8b8296a","Type":"ContainerDied","Data":"ff3cda574ffa06b569ec59c5c787faeebcae87accc871160372f65a18546b319"} Oct 30 21:34:07.107341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.106838 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff3cda574ffa06b569ec59c5c787faeebcae87accc871160372f65a18546b319" Oct 30 21:34:07.255719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.255695 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:07.298307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.298275 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e209c61-424e-4070-8358-6d4bc95ab185-kube-api-access-z27vt\" (UniqueName: \"kubernetes.io/projected/0e209c61-424e-4070-8358-6d4bc95ab185-kube-api-access-z27vt\") pod \"0e209c61-424e-4070-8358-6d4bc95ab185\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " Oct 30 21:34:07.298456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.298427 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-openshift-ca\") pod \"0e209c61-424e-4070-8358-6d4bc95ab185\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " Oct 30 21:34:07.298542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.298476 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-xks-ca\") pod \"0e209c61-424e-4070-8358-6d4bc95ab185\" (UID: \"0e209c61-424e-4070-8358-6d4bc95ab185\") " Oct 30 21:34:07.298811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.298788 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-openshift-ca" pod "0e209c61-424e-4070-8358-6d4bc95ab185" (UID: "0e209c61-424e-4070-8358-6d4bc95ab185"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:07.300774 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.300752 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e209c61-424e-4070-8358-6d4bc95ab185-kube-api-access-z27vt" pod "0e209c61-424e-4070-8358-6d4bc95ab185" (UID: "0e209c61-424e-4070-8358-6d4bc95ab185"). InnerVolumeSpecName "kube-api-access-z27vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:07.300864 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.300767 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-xks-ca" pod "0e209c61-424e-4070-8358-6d4bc95ab185" (UID: "0e209c61-424e-4070-8358-6d4bc95ab185"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:07.400277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.400198 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:07.400277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.400228 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e209c61-424e-4070-8358-6d4bc95ab185-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:07.400277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:07.400239 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z27vt\" (UniqueName: \"kubernetes.io/projected/0e209c61-424e-4070-8358-6d4bc95ab185-kube-api-access-z27vt\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:08.114011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:08.113986 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" Oct 30 21:34:08.114433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:08.114023 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz" event={"ID":"0e209c61-424e-4070-8358-6d4bc95ab185","Type":"ContainerDied","Data":"80ccb72b56661f7c27ac0fcaf664ec9a89ed22f03b47efc73c1e81fe60079ccf"} Oct 30 21:34:08.114433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:08.114069 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ccb72b56661f7c27ac0fcaf664ec9a89ed22f03b47efc73c1e81fe60079ccf" Oct 30 21:34:12.477925 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:34:12.477889 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:34:13.616811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.616777 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph"] Oct 30 21:34:13.617286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.617272 4526 cpu_manager.go:414] "containerMap: removing container" podUID="bc159488-381b-4c46-8add-de64f8b8296a" containerName="revoke-keys" Oct 30 21:34:13.617324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.617287 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc159488-381b-4c46-8add-de64f8b8296a" containerName="revoke-keys" Oct 30 21:34:13.617324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.617308 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0e209c61-424e-4070-8358-6d4bc95ab185" containerName="revoke-keys" Oct 30 21:34:13.617324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.617314 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e209c61-424e-4070-8358-6d4bc95ab185" containerName="revoke-keys" Oct 30 21:34:13.617435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.617407 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e209c61-424e-4070-8358-6d4bc95ab185" containerName="revoke-keys" Oct 30 21:34:13.617435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.617423 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc159488-381b-4c46-8add-de64f8b8296a" containerName="revoke-keys" Oct 30 21:34:13.621037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.621022 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.670546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.670483 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkthk\" (UniqueName: \"kubernetes.io/projected/34621233-47c6-4dd8-9952-dc3978591643-kube-api-access-gkthk\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.670733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.670712 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34621233-47c6-4dd8-9952-dc3978591643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.670808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.670774 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34621233-47c6-4dd8-9952-dc3978591643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.677862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.677834 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph"] Oct 30 21:34:13.723105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.723051 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:34:13.723105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.723104 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:34:13.723354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.723167 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:34:13.723354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.723273 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:34:13.772327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.772296 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34621233-47c6-4dd8-9952-dc3978591643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.772473 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.772338 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkthk\" (UniqueName: \"kubernetes.io/projected/34621233-47c6-4dd8-9952-dc3978591643-kube-api-access-gkthk\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.772554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.772533 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34621233-47c6-4dd8-9952-dc3978591643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.773129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.773106 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34621233-47c6-4dd8-9952-dc3978591643-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.774819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.774803 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34621233-47c6-4dd8-9952-dc3978591643-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.829741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.829709 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkthk\" (UniqueName: \"kubernetes.io/projected/34621233-47c6-4dd8-9952-dc3978591643-kube-api-access-gkthk\") pod \"maas-api-revoke-sub-29a43e7f-22wph\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:13.934067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:13.933976 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:14.312698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:14.312674 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph"] Oct 30 21:34:14.314182 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:34:14.314146 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34621233_47c6_4dd8_9952_dc3978591643.slice/crio-a6bf4d742aea98f11cd7c5ff07522122608309c45aba0f4725f4e9a3e0eafedf WatchSource:0}: Error finding container a6bf4d742aea98f11cd7c5ff07522122608309c45aba0f4725f4e9a3e0eafedf: Status 404 returned error can't find the container with id a6bf4d742aea98f11cd7c5ff07522122608309c45aba0f4725f4e9a3e0eafedf Oct 30 21:34:15.159321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:15.159282 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" event={"ID":"34621233-47c6-4dd8-9952-dc3978591643","Type":"ContainerStarted","Data":"dd554a15dbd5ae850fdeb3f108d53e0d632877de933fafe6fa77a1362af75056"} Oct 30 21:34:15.159753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:15.159349 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" event={"ID":"34621233-47c6-4dd8-9952-dc3978591643","Type":"ContainerStarted","Data":"a6bf4d742aea98f11cd7c5ff07522122608309c45aba0f4725f4e9a3e0eafedf"} Oct 30 21:34:15.275584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:15.275532 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" podStartSLOduration=2.275515599 podStartE2EDuration="2.275515599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:15.272584791 +0000 UTC m=+1045.405957988" watchObservedRunningTime="2026-10-30 21:34:15.275515599 +0000 UTC m=+1045.408888796" Oct 30 21:34:16.167538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:16.167507 4526 generic.go:356] "Generic (PLEG): container finished" podID="34621233-47c6-4dd8-9952-dc3978591643" containerID="dd554a15dbd5ae850fdeb3f108d53e0d632877de933fafe6fa77a1362af75056" exitCode=0 Oct 30 21:34:16.167978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:16.167552 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" event={"ID":"34621233-47c6-4dd8-9952-dc3978591643","Type":"ContainerDied","Data":"dd554a15dbd5ae850fdeb3f108d53e0d632877de933fafe6fa77a1362af75056"} Oct 30 21:34:17.326466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.326435 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:17.413904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.413871 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34621233-47c6-4dd8-9952-dc3978591643-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34621233-47c6-4dd8-9952-dc3978591643-maas-api-openshift-ca\") pod \"34621233-47c6-4dd8-9952-dc3978591643\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " Oct 30 21:34:17.414089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.413930 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34621233-47c6-4dd8-9952-dc3978591643-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34621233-47c6-4dd8-9952-dc3978591643-maas-api-xks-ca\") pod \"34621233-47c6-4dd8-9952-dc3978591643\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " Oct 30 21:34:17.414146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.414127 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34621233-47c6-4dd8-9952-dc3978591643-kube-api-access-gkthk\" (UniqueName: \"kubernetes.io/projected/34621233-47c6-4dd8-9952-dc3978591643-kube-api-access-gkthk\") pod \"34621233-47c6-4dd8-9952-dc3978591643\" (UID: \"34621233-47c6-4dd8-9952-dc3978591643\") " Oct 30 21:34:17.414312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.414287 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34621233-47c6-4dd8-9952-dc3978591643-maas-api-openshift-ca" pod "34621233-47c6-4dd8-9952-dc3978591643" (UID: "34621233-47c6-4dd8-9952-dc3978591643"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:17.414644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.414602 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34621233-47c6-4dd8-9952-dc3978591643-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:17.416267 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.416235 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34621233-47c6-4dd8-9952-dc3978591643-maas-api-xks-ca" pod "34621233-47c6-4dd8-9952-dc3978591643" (UID: "34621233-47c6-4dd8-9952-dc3978591643"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:17.416370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.416351 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34621233-47c6-4dd8-9952-dc3978591643-kube-api-access-gkthk" pod "34621233-47c6-4dd8-9952-dc3978591643" (UID: "34621233-47c6-4dd8-9952-dc3978591643"). InnerVolumeSpecName "kube-api-access-gkthk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:17.515435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.515401 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34621233-47c6-4dd8-9952-dc3978591643-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:17.515435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.515435 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkthk\" (UniqueName: \"kubernetes.io/projected/34621233-47c6-4dd8-9952-dc3978591643-kube-api-access-gkthk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:17.818372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.818329 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8"] Oct 30 21:34:17.819147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.819128 4526 cpu_manager.go:414] "containerMap: removing container" podUID="34621233-47c6-4dd8-9952-dc3978591643" containerName="revoke-keys" Oct 30 21:34:17.819200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.819152 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34621233-47c6-4dd8-9952-dc3978591643" containerName="revoke-keys" Oct 30 21:34:17.819330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.819317 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="34621233-47c6-4dd8-9952-dc3978591643" containerName="revoke-keys" Oct 30 21:34:17.823317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.823301 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:17.879368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.879337 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8"] Oct 30 21:34:17.919866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.919826 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:17.920045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.919954 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfg4\" (UniqueName: \"kubernetes.io/projected/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-kube-api-access-jrfg4\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:17.920045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:17.919994 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:18.021601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.021566 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfg4\" (UniqueName: \"kubernetes.io/projected/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-kube-api-access-jrfg4\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:18.021601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.021607 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:18.021845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.021689 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:18.022344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.022307 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:18.024260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.024232 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:18.078235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.078165 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfg4\" (UniqueName: \"kubernetes.io/projected/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-kube-api-access-jrfg4\") pod \"maas-api-revoke-sub-76e90454-dndn8\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:18.138500 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.138465 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:18.195054 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.195019 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" event={"ID":"34621233-47c6-4dd8-9952-dc3978591643","Type":"ContainerDied","Data":"a6bf4d742aea98f11cd7c5ff07522122608309c45aba0f4725f4e9a3e0eafedf"} Oct 30 21:34:18.195199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.195075 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph" Oct 30 21:34:18.195199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.195132 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6bf4d742aea98f11cd7c5ff07522122608309c45aba0f4725f4e9a3e0eafedf" Oct 30 21:34:18.514062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:18.514031 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8"] Oct 30 21:34:18.515187 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:34:18.515157 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb90b91_6a9c_4822_a44d_31ab10ec0b8f.slice/crio-4720d010da5d617eaf3d759b3f8e03ee01d0e0895099c8116c767d245b7cc8eb WatchSource:0}: Error finding container 4720d010da5d617eaf3d759b3f8e03ee01d0e0895099c8116c767d245b7cc8eb: Status 404 returned error can't find the container with id 4720d010da5d617eaf3d759b3f8e03ee01d0e0895099c8116c767d245b7cc8eb Oct 30 21:34:19.204145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.204064 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" event={"ID":"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f","Type":"ContainerStarted","Data":"f33df8ee237feedb22639768b2eb34bf847351e3ff244ffd5af5b603503739b9"} Oct 30 21:34:19.204145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.204124 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" event={"ID":"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f","Type":"ContainerStarted","Data":"4720d010da5d617eaf3d759b3f8e03ee01d0e0895099c8116c767d245b7cc8eb"} Oct 30 21:34:19.317734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.317684 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" podStartSLOduration=2.317667216 podStartE2EDuration="2.317667216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:19.315791094 +0000 UTC m=+1049.449164292" watchObservedRunningTime="2026-10-30 21:34:19.317667216 +0000 UTC m=+1049.451040413" Oct 30 21:34:19.553604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.553568 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc"] Oct 30 21:34:19.557438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.557422 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.615198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.615164 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc"] Oct 30 21:34:19.639060 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.639019 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hfw2\" (UniqueName: \"kubernetes.io/projected/3ceda648-9305-4c3c-b6ab-613e87c1fa57-kube-api-access-5hfw2\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.639233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.639121 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.639332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.639312 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.740224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.740182 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.740224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.740229 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfw2\" (UniqueName: \"kubernetes.io/projected/3ceda648-9305-4c3c-b6ab-613e87c1fa57-kube-api-access-5hfw2\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.740476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.740283 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.740882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.740855 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.742824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.742802 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.796714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.796680 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hfw2\" (UniqueName: \"kubernetes.io/projected/3ceda648-9305-4c3c-b6ab-613e87c1fa57-kube-api-access-5hfw2\") pod \"maas-api-revoke-sub-6575b78a-rvgdc\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:19.870704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:19.870622 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:20.213384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:20.213291 4526 generic.go:356] "Generic (PLEG): container finished" podID="ffb90b91-6a9c-4822-a44d-31ab10ec0b8f" containerID="f33df8ee237feedb22639768b2eb34bf847351e3ff244ffd5af5b603503739b9" exitCode=0 Oct 30 21:34:20.213595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:20.213373 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" event={"ID":"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f","Type":"ContainerDied","Data":"f33df8ee237feedb22639768b2eb34bf847351e3ff244ffd5af5b603503739b9"} Oct 30 21:34:20.252329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:20.252306 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc"] Oct 30 21:34:20.254184 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:34:20.254160 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ceda648_9305_4c3c_b6ab_613e87c1fa57.slice/crio-ea15519d1eb347e98d42cc881d6d923a701b19f4769b83211f35713fafd992ee WatchSource:0}: Error finding container ea15519d1eb347e98d42cc881d6d923a701b19f4769b83211f35713fafd992ee: Status 404 returned error can't find the container with id ea15519d1eb347e98d42cc881d6d923a701b19f4769b83211f35713fafd992ee Oct 30 21:34:21.221454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.221416 4526 generic.go:356] "Generic (PLEG): container finished" podID="3ceda648-9305-4c3c-b6ab-613e87c1fa57" containerID="c9520b3ed8fe0a035bbe81e96238fcda28c39b25ea2f91505c136cfd157d0d0c" exitCode=0 Oct 30 21:34:21.221454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.221454 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" event={"ID":"3ceda648-9305-4c3c-b6ab-613e87c1fa57","Type":"ContainerDied","Data":"c9520b3ed8fe0a035bbe81e96238fcda28c39b25ea2f91505c136cfd157d0d0c"} Oct 30 21:34:21.221947 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.221524 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" event={"ID":"3ceda648-9305-4c3c-b6ab-613e87c1fa57","Type":"ContainerStarted","Data":"ea15519d1eb347e98d42cc881d6d923a701b19f4769b83211f35713fafd992ee"} Oct 30 21:34:21.370370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.370343 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:21.457381 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.457350 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-openshift-ca\") pod \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " Oct 30 21:34:21.457584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.457413 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-kube-api-access-jrfg4\" (UniqueName: \"kubernetes.io/projected/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-kube-api-access-jrfg4\") pod \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " Oct 30 21:34:21.457584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.457548 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-xks-ca\") pod \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\" (UID: \"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f\") " Oct 30 21:34:21.457782 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.457759 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-openshift-ca" pod "ffb90b91-6a9c-4822-a44d-31ab10ec0b8f" (UID: "ffb90b91-6a9c-4822-a44d-31ab10ec0b8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:21.458016 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.457999 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:21.459422 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.459403 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-kube-api-access-jrfg4" pod "ffb90b91-6a9c-4822-a44d-31ab10ec0b8f" (UID: "ffb90b91-6a9c-4822-a44d-31ab10ec0b8f"). InnerVolumeSpecName "kube-api-access-jrfg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:21.459537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.459522 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-xks-ca" pod "ffb90b91-6a9c-4822-a44d-31ab10ec0b8f" (UID: "ffb90b91-6a9c-4822-a44d-31ab10ec0b8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:21.559655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.559620 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrfg4\" (UniqueName: \"kubernetes.io/projected/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-kube-api-access-jrfg4\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:21.559655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:21.559653 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:22.230428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.230399 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" Oct 30 21:34:22.230903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.230442 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8" event={"ID":"ffb90b91-6a9c-4822-a44d-31ab10ec0b8f","Type":"ContainerDied","Data":"4720d010da5d617eaf3d759b3f8e03ee01d0e0895099c8116c767d245b7cc8eb"} Oct 30 21:34:22.230903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.230535 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4720d010da5d617eaf3d759b3f8e03ee01d0e0895099c8116c767d245b7cc8eb" Oct 30 21:34:22.387111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.387082 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:22.470674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.470644 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ceda648-9305-4c3c-b6ab-613e87c1fa57-kube-api-access-5hfw2\" (UniqueName: \"kubernetes.io/projected/3ceda648-9305-4c3c-b6ab-613e87c1fa57-kube-api-access-5hfw2\") pod \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " Oct 30 21:34:22.470878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.470834 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-openshift-ca\") pod \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " Oct 30 21:34:22.470878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.470859 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-xks-ca\") pod \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\" (UID: \"3ceda648-9305-4c3c-b6ab-613e87c1fa57\") " Oct 30 21:34:22.471209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.471177 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-openshift-ca" pod "3ceda648-9305-4c3c-b6ab-613e87c1fa57" (UID: "3ceda648-9305-4c3c-b6ab-613e87c1fa57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:22.472958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.472938 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ceda648-9305-4c3c-b6ab-613e87c1fa57-kube-api-access-5hfw2" pod "3ceda648-9305-4c3c-b6ab-613e87c1fa57" (UID: "3ceda648-9305-4c3c-b6ab-613e87c1fa57"). InnerVolumeSpecName "kube-api-access-5hfw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:22.473190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.473172 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-xks-ca" pod "3ceda648-9305-4c3c-b6ab-613e87c1fa57" (UID: "3ceda648-9305-4c3c-b6ab-613e87c1fa57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:22.572760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.572681 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:22.572760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.572707 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ceda648-9305-4c3c-b6ab-613e87c1fa57-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:22.572760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:22.572717 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hfw2\" (UniqueName: \"kubernetes.io/projected/3ceda648-9305-4c3c-b6ab-613e87c1fa57-kube-api-access-5hfw2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:23.238159 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:23.238129 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" Oct 30 21:34:23.238609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:23.238167 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc" event={"ID":"3ceda648-9305-4c3c-b6ab-613e87c1fa57","Type":"ContainerDied","Data":"ea15519d1eb347e98d42cc881d6d923a701b19f4769b83211f35713fafd992ee"} Oct 30 21:34:23.238609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:23.238222 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea15519d1eb347e98d42cc881d6d923a701b19f4769b83211f35713fafd992ee" Oct 30 21:34:28.214577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:28.214537 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podUID="a4207178a950303c256d4f333088d96f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:34:29.284657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.284617 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d"] Oct 30 21:34:29.285150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.285137 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3ceda648-9305-4c3c-b6ab-613e87c1fa57" containerName="revoke-keys" Oct 30 21:34:29.285194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.285151 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ceda648-9305-4c3c-b6ab-613e87c1fa57" containerName="revoke-keys" Oct 30 21:34:29.285194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.285173 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ffb90b91-6a9c-4822-a44d-31ab10ec0b8f" containerName="revoke-keys" Oct 30 21:34:29.285194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.285179 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb90b91-6a9c-4822-a44d-31ab10ec0b8f" containerName="revoke-keys" Oct 30 21:34:29.285291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.285251 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ceda648-9305-4c3c-b6ab-613e87c1fa57" containerName="revoke-keys" Oct 30 21:34:29.285291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.285263 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb90b91-6a9c-4822-a44d-31ab10ec0b8f" containerName="revoke-keys" Oct 30 21:34:29.289105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.289089 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.391282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.391246 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:34:29.391507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.391307 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:34:29.391507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.391321 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:34:29.391507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.391347 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:34:29.395030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.395004 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d"] Oct 30 21:34:29.449900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.449861 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zrk\" (UniqueName: \"kubernetes.io/projected/3d0302e6-03e3-418f-b6e7-43a485ec5537-kube-api-access-s7zrk\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.449900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.449905 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.450110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.449930 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.551367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.551270 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zrk\" (UniqueName: \"kubernetes.io/projected/3d0302e6-03e3-418f-b6e7-43a485ec5537-kube-api-access-s7zrk\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.551367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.551313 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.551367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.551338 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.552048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.552028 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.553871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.553851 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.607391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.607359 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zrk\" (UniqueName: \"kubernetes.io/projected/3d0302e6-03e3-418f-b6e7-43a485ec5537-kube-api-access-s7zrk\") pod \"maas-api-revoke-sub-12facb17-gzp9d\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:29.901781 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:29.901698 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:30.279408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:30.279379 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d"] Oct 30 21:34:30.281120 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:34:30.281083 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0302e6_03e3_418f_b6e7_43a485ec5537.slice/crio-7af6ddb5e4a04c3edf7b78284dcc50d95396a280bd92a1f03b474acceaacce89 WatchSource:0}: Error finding container 7af6ddb5e4a04c3edf7b78284dcc50d95396a280bd92a1f03b474acceaacce89: Status 404 returned error can't find the container with id 7af6ddb5e4a04c3edf7b78284dcc50d95396a280bd92a1f03b474acceaacce89 Oct 30 21:34:31.292624 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:31.292590 4526 generic.go:356] "Generic (PLEG): container finished" podID="3d0302e6-03e3-418f-b6e7-43a485ec5537" containerID="92cbcab3e8c44661c62f1cadcceb24f2117015c9b847b647853d7ebeb17c90c5" exitCode=0 Oct 30 21:34:31.293048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:31.292672 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" event={"ID":"3d0302e6-03e3-418f-b6e7-43a485ec5537","Type":"ContainerDied","Data":"92cbcab3e8c44661c62f1cadcceb24f2117015c9b847b647853d7ebeb17c90c5"} Oct 30 21:34:31.293048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:31.292729 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" event={"ID":"3d0302e6-03e3-418f-b6e7-43a485ec5537","Type":"ContainerStarted","Data":"7af6ddb5e4a04c3edf7b78284dcc50d95396a280bd92a1f03b474acceaacce89"} Oct 30 21:34:32.438153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.438129 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:32.481081 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.481050 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0302e6-03e3-418f-b6e7-43a485ec5537-kube-api-access-s7zrk\" (UniqueName: \"kubernetes.io/projected/3d0302e6-03e3-418f-b6e7-43a485ec5537-kube-api-access-s7zrk\") pod \"3d0302e6-03e3-418f-b6e7-43a485ec5537\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " Oct 30 21:34:32.481252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.481088 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-xks-ca\") pod \"3d0302e6-03e3-418f-b6e7-43a485ec5537\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " Oct 30 21:34:32.481252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.481174 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-openshift-ca\") pod \"3d0302e6-03e3-418f-b6e7-43a485ec5537\" (UID: \"3d0302e6-03e3-418f-b6e7-43a485ec5537\") " Oct 30 21:34:32.481594 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.481562 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-openshift-ca" pod "3d0302e6-03e3-418f-b6e7-43a485ec5537" (UID: "3d0302e6-03e3-418f-b6e7-43a485ec5537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:32.483448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.483424 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-xks-ca" pod "3d0302e6-03e3-418f-b6e7-43a485ec5537" (UID: "3d0302e6-03e3-418f-b6e7-43a485ec5537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:32.483559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.483464 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0302e6-03e3-418f-b6e7-43a485ec5537-kube-api-access-s7zrk" pod "3d0302e6-03e3-418f-b6e7-43a485ec5537" (UID: "3d0302e6-03e3-418f-b6e7-43a485ec5537"). InnerVolumeSpecName "kube-api-access-s7zrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:32.582865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.582779 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7zrk\" (UniqueName: \"kubernetes.io/projected/3d0302e6-03e3-418f-b6e7-43a485ec5537-kube-api-access-s7zrk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:32.582865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.582807 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:32.582865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:32.582820 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d0302e6-03e3-418f-b6e7-43a485ec5537-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:33.307092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:33.307054 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" event={"ID":"3d0302e6-03e3-418f-b6e7-43a485ec5537","Type":"ContainerDied","Data":"7af6ddb5e4a04c3edf7b78284dcc50d95396a280bd92a1f03b474acceaacce89"} Oct 30 21:34:33.307092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:33.307086 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d" Oct 30 21:34:33.307411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:33.307112 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7af6ddb5e4a04c3edf7b78284dcc50d95396a280bd92a1f03b474acceaacce89" Oct 30 21:34:42.478027 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:34:42.477994 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:34:51.332968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.332935 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f"] Oct 30 21:34:51.333447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.333434 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3d0302e6-03e3-418f-b6e7-43a485ec5537" containerName="revoke-keys" Oct 30 21:34:51.333509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.333448 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0302e6-03e3-418f-b6e7-43a485ec5537" containerName="revoke-keys" Oct 30 21:34:51.333553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.333547 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0302e6-03e3-418f-b6e7-43a485ec5537" containerName="revoke-keys" Oct 30 21:34:51.337097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.337083 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.388932 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.388897 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e4dadd-openshift-default-dockercfg-vmhcw\"" Oct 30 21:34:51.443787 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.443753 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f"] Oct 30 21:34:51.490641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.490603 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.490641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.490643 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxs6j\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-kube-api-access-lxs6j\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.490909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.490733 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.490909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.490779 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.490909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.490866 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.491045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.490910 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.491086 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.491048 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.491086 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.491075 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.491195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.491161 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592473 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592561 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592580 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxs6j\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-kube-api-access-lxs6j\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592601 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592620 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592660 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592694 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592806 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.592873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.592832 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.593179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.593165 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.593269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.593243 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.593448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.593274 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.593548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.593337 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.593548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.593470 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.595644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.595627 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.595795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.595780 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.652091 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.652051 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.652245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.652220 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxs6j\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-kube-api-access-lxs6j\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:51.951233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:51.951123 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:52.383117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:52.383090 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f"] Oct 30 21:34:52.383725 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:34:52.383698 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ec57f7_0478_4d5c_8189_a4e4c6168a2c.slice/crio-31a6ab997224eebc3b8e12d0e92b9b2f7107ac8d300e1c8ac87bcb6bf870f68e WatchSource:0}: Error finding container 31a6ab997224eebc3b8e12d0e92b9b2f7107ac8d300e1c8ac87bcb6bf870f68e: Status 404 returned error can't find the container with id 31a6ab997224eebc3b8e12d0e92b9b2f7107ac8d300e1c8ac87bcb6bf870f68e Oct 30 21:34:52.386224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:52.386190 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:34:52.386337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:52.386266 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:34:52.386337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:52.386312 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:34:52.431847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:52.431808 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" event={"ID":"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c","Type":"ContainerStarted","Data":"31a6ab997224eebc3b8e12d0e92b9b2f7107ac8d300e1c8ac87bcb6bf870f68e"} Oct 30 21:34:53.440015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:53.439971 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" event={"ID":"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c","Type":"ContainerStarted","Data":"87d43f7443f708813824a56a0113174b6a5b581b76eb8c41638af187fa8ae5e8"} Oct 30 21:34:53.621807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:53.621751 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podStartSLOduration=2.62173247 podStartE2EDuration="2.62173247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:53.620159142 +0000 UTC m=+1083.753532340" watchObservedRunningTime="2026-10-30 21:34:53.62173247 +0000 UTC m=+1083.755105673" Oct 30 21:34:53.952255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:53.952220 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:53.953754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:53.953724 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:34:53.953906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:53.953787 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:34:54.951616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:54.951580 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:34:54.952097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:54.951633 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:34:55.951597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:55.951565 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:34:55.951774 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:55.951619 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:34:56.952380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:56.952345 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:34:56.952851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:56.952401 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:34:57.951641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:57.951607 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 21:34:57.951820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:57.951660 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 21:34:59.103347 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:59.103314 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:59.103886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:59.103812 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:34:59.105195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:34:59.105170 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:35:03.593105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:03.593070 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4"] Oct 30 21:35:03.593590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:03.593302 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" containerID="cri-o://ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7" gracePeriod=30 Oct 30 21:35:08.854004 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.853978 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:35:08.897029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.896938 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-socket\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.896986 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-token\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-token\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897015 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-credential-socket\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897073 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-certs\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897104 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istiod-ca-cert\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897145 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-envoy\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897217 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-podinfo\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897267 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-socket" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:08.897290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897277 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-kube-api-access-hnhn7\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-kube-api-access-hnhn7\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897362 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-data\") pod \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\" (UID: \"66cdf6fb-6c69-4ad3-b49c-8388305fed43\") " Oct 30 21:35:08.897641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897396 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-credential-socket" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:08.897641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897547 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-certs" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:08.897641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897621 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istiod-ca-cert" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:08.897866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897745 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-data" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:08.897930 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897879 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:08.897930 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897897 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:08.897930 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897913 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:08.898249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.897968 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:08.898249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.898079 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:08.899557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.899532 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-token" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:08.899666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.899565 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-kube-api-access-hnhn7" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "kube-api-access-hnhn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:08.899666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.899597 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-podinfo" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:35:08.899666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.899637 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-envoy" pod "66cdf6fb-6c69-4ad3-b49c-8388305fed43" (UID: "66cdf6fb-6c69-4ad3-b49c-8388305fed43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:08.999483 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.999449 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:08.999483 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.999476 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:08.999483 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.999509 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cdf6fb-6c69-4ad3-b49c-8388305fed43-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:08.999731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:08.999518 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnhn7\" (UniqueName: \"kubernetes.io/projected/66cdf6fb-6c69-4ad3-b49c-8388305fed43-kube-api-access-hnhn7\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:35:09.397887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.397851 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4"] Oct 30 21:35:09.398365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.398352 4526 cpu_manager.go:414] "containerMap: removing container" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" Oct 30 21:35:09.398423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.398366 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" Oct 30 21:35:09.398523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.398512 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerName="istio-proxy" Oct 30 21:35:09.402209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.402192 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.456883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.456848 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9796f1\"/\"distinct-w3-9796f1-kserve-self-signed-certs\"" Oct 30 21:35:09.460203 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.460171 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4"] Oct 30 21:35:09.504183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.504139 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-home\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.504360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.504342 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-tmp-dir\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.504408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.504373 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec879707-8027-4f91-8faf-a9b97e4a1267-tls-certs\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.504408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.504396 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzt85\" (UniqueName: \"kubernetes.io/projected/ec879707-8027-4f91-8faf-a9b97e4a1267-kube-api-access-nzt85\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.504571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.504544 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-dshm\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.504638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.504589 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-model-cache\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.556134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.556068 4526 generic.go:356] "Generic (PLEG): container finished" podID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" containerID="ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7" exitCode=0 Oct 30 21:35:09.556134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.556133 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" event={"ID":"66cdf6fb-6c69-4ad3-b49c-8388305fed43","Type":"ContainerDied","Data":"ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7"} Oct 30 21:35:09.556409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.556185 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" event={"ID":"66cdf6fb-6c69-4ad3-b49c-8388305fed43","Type":"ContainerDied","Data":"bbac10b3513f64f86c95e774513aaae986a928dd852c374165987029134d79d8"} Oct 30 21:35:09.556409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.556213 4526 scope.go:122] "RemoveContainer" containerID="ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7" Oct 30 21:35:09.556409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.556362 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4" Oct 30 21:35:09.567807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.567787 4526 scope.go:122] "RemoveContainer" containerID="ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7" Oct 30 21:35:09.568130 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:35:09.568111 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7\": container with ID starting with ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7 not found: ID does not exist" containerID="ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7" Oct 30 21:35:09.568178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.568139 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7"} err="failed to get container status \"ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7\": rpc error: code = NotFound desc = could not find container \"ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7\": container with ID starting with ad5fe38e6bcbcb0821c388e36d183572964e16065ce5b7710f83784cc354c9b7 not found: ID does not exist" Oct 30 21:35:09.605879 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.605844 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-home\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.606009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.605993 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-tmp-dir\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.606050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.606017 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec879707-8027-4f91-8faf-a9b97e4a1267-tls-certs\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.606050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.606039 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzt85\" (UniqueName: \"kubernetes.io/projected/ec879707-8027-4f91-8faf-a9b97e4a1267-kube-api-access-nzt85\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.606125 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.606061 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-dshm\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.606125 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.606091 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-model-cache\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.606247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.606227 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-home\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.606433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.606413 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-tmp-dir\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.606512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.606443 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-model-cache\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.608482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.608456 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-dshm\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.608791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.608775 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec879707-8027-4f91-8faf-a9b97e4a1267-tls-certs\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.664243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.664164 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzt85\" (UniqueName: \"kubernetes.io/projected/ec879707-8027-4f91-8faf-a9b97e4a1267-kube-api-access-nzt85\") pod \"distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.714820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.714788 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:09.722625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.722600 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4"] Oct 30 21:35:09.725664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.725635 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tzxd4"] Oct 30 21:35:09.988564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:09.987546 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66cdf6fb-6c69-4ad3-b49c-8388305fed43" path="/var/lib/kubelet/pods/66cdf6fb-6c69-4ad3-b49c-8388305fed43/volumes" Oct 30 21:35:10.095006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:10.094974 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4"] Oct 30 21:35:10.095257 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:35:10.095235 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec879707_8027_4f91_8faf_a9b97e4a1267.slice/crio-0799686922a30e8a3cc59529368ee310210959bec50dbde614f4246a6e12b3b1 WatchSource:0}: Error finding container 0799686922a30e8a3cc59529368ee310210959bec50dbde614f4246a6e12b3b1: Status 404 returned error can't find the container with id 0799686922a30e8a3cc59529368ee310210959bec50dbde614f4246a6e12b3b1 Oct 30 21:35:10.564479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:10.564443 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" event={"ID":"ec879707-8027-4f91-8faf-a9b97e4a1267","Type":"ContainerStarted","Data":"0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023"} Oct 30 21:35:10.564706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:10.564483 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" event={"ID":"ec879707-8027-4f91-8faf-a9b97e4a1267","Type":"ContainerStarted","Data":"0799686922a30e8a3cc59529368ee310210959bec50dbde614f4246a6e12b3b1"} Oct 30 21:35:10.689610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:10.689557 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podStartSLOduration=1.6895431109999999 podStartE2EDuration="1.689543111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:10.688520655 +0000 UTC m=+1100.821893854" watchObservedRunningTime="2026-10-30 21:35:10.689543111 +0000 UTC m=+1100.822916320" Oct 30 21:35:10.714892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:10.714857 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:12.479059 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:35:12.479018 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:35:19.715970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:19.715935 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:19.728918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:19.728891 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:20.641504 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:20.641463 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:35:26.002718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.002683 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9lshl-w29xw"] Oct 30 21:35:26.006706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.006690 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.109979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.109936 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k4xtg\"" Oct 30 21:35:26.109979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.109981 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:35:26.113550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.113524 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9lshl-w29xw"] Oct 30 21:35:26.199940 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.199909 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-trusted-ca-bundle\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.200147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.199961 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcq8g\" (UniqueName: \"kubernetes.io/projected/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-kube-api-access-dcq8g\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.200147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.200118 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-service-ca-bundle\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.200258 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.200223 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-archives-path\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.210594 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.210572 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:35:26.301389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.301362 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-service-ca-bundle\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.301582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.301420 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-archives-path\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.301582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.301479 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-trusted-ca-bundle\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.301582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.301565 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcq8g\" (UniqueName: \"kubernetes.io/projected/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-kube-api-access-dcq8g\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.301962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.301941 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-archives-path\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.302118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.302100 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-service-ca-bundle\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.302297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.302277 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-trusted-ca-bundle\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.358295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.358260 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcq8g\" (UniqueName: \"kubernetes.io/projected/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-kube-api-access-dcq8g\") pod \"periodic-gathering-9lshl-w29xw\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.620461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.620373 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:35:26.997899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:26.997872 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9lshl-w29xw"] Oct 30 21:35:26.998629 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:35:26.998607 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46d54e3_d4cc_4413_8fb1_0610b62b1b4c.slice/crio-c8eaf2ae051c58082bd75ec6e5d861527c2baae18244d290cf022685a907b6f1 WatchSource:0}: Error finding container c8eaf2ae051c58082bd75ec6e5d861527c2baae18244d290cf022685a907b6f1: Status 404 returned error can't find the container with id c8eaf2ae051c58082bd75ec6e5d861527c2baae18244d290cf022685a907b6f1 Oct 30 21:35:27.678465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:27.678427 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9lshl-w29xw" event={"ID":"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c","Type":"ContainerStarted","Data":"16adf9a62e7e47946363907cb9bcd38131faf0874f8dafd8fa2d291e79438e29"} Oct 30 21:35:27.678918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:27.678479 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9lshl-w29xw" event={"ID":"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c","Type":"ContainerStarted","Data":"c8eaf2ae051c58082bd75ec6e5d861527c2baae18244d290cf022685a907b6f1"} Oct 30 21:35:27.807853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:27.807804 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-9lshl-w29xw" podStartSLOduration=2.807788313 podStartE2EDuration="2.807788313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:27.804869112 +0000 UTC m=+1117.938242310" watchObservedRunningTime="2026-10-30 21:35:27.807788313 +0000 UTC m=+1117.941161510" Oct 30 21:35:35.462004 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.461914 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg"] Oct 30 21:35:35.466200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.466184 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.498136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.498094 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-dshm\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.498330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.498215 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zht4v\" (UniqueName: \"kubernetes.io/projected/ba054402-cbd1-4b77-8016-8bb37af9a9ae-kube-api-access-zht4v\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.498330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.498240 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tmp-dir\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.498330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.498329 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-home\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.498549 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.498355 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tls-certs\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.498549 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.498400 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-model-cache\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.568070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.568027 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-021907\"/\"default-dockercfg-qzw2w\"" Oct 30 21:35:35.568251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.568089 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-021907\"/\"kube-root-ca.crt\"" Oct 30 21:35:35.568251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.568089 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-021907\"/\"openshift-service-ca.crt\"" Oct 30 21:35:35.568251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.568128 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-021907\"/\"distinct-w5-021907-kserve-self-signed-certs\"" Oct 30 21:35:35.569523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.569482 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg"] Oct 30 21:35:35.601232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.601158 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zht4v\" (UniqueName: \"kubernetes.io/projected/ba054402-cbd1-4b77-8016-8bb37af9a9ae-kube-api-access-zht4v\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.601232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.601221 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tmp-dir\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.601784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.601308 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-home\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.601784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.601349 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tls-certs\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.601784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.601410 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-model-cache\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.601784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.601478 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-dshm\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.602444 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.602420 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-home\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.603333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.603314 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tmp-dir\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.603523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.603470 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-model-cache\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.607679 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.607654 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tls-certs\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.609074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.609041 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-dshm\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.659186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.659157 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zht4v\" (UniqueName: \"kubernetes.io/projected/ba054402-cbd1-4b77-8016-8bb37af9a9ae-kube-api-access-zht4v\") pod \"distinct-w5-021907-kserve-785c74b5cc-zrvxg\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:35.779135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:35.779044 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:36.155961 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:36.155928 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg"] Oct 30 21:35:36.156810 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:35:36.156783 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba054402_cbd1_4b77_8016_8bb37af9a9ae.slice/crio-90845f2f64cc0008cda32e781694303b5d9683e18d1310aba0d9b09aadc97c0e WatchSource:0}: Error finding container 90845f2f64cc0008cda32e781694303b5d9683e18d1310aba0d9b09aadc97c0e: Status 404 returned error can't find the container with id 90845f2f64cc0008cda32e781694303b5d9683e18d1310aba0d9b09aadc97c0e Oct 30 21:35:36.739317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:36.739274 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" event={"ID":"ba054402-cbd1-4b77-8016-8bb37af9a9ae","Type":"ContainerStarted","Data":"f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67"} Oct 30 21:35:36.739793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:36.739327 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" event={"ID":"ba054402-cbd1-4b77-8016-8bb37af9a9ae","Type":"ContainerStarted","Data":"90845f2f64cc0008cda32e781694303b5d9683e18d1310aba0d9b09aadc97c0e"} Oct 30 21:35:36.779816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:36.779787 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:36.857313 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:36.857259 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podStartSLOduration=1.857243875 podStartE2EDuration="1.857243875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:36.854509886 +0000 UTC m=+1126.987883085" watchObservedRunningTime="2026-10-30 21:35:36.857243875 +0000 UTC m=+1126.990617073" Oct 30 21:35:42.479433 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:35:42.479393 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:35:45.779938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:45.779903 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:45.792411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:45.792385 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:45.809192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:45.809155 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:35:51.127869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.127836 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp"] Oct 30 21:35:51.137457 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.136551 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.189595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.189558 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp"] Oct 30 21:35:51.234564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.234531 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"distinct-2-w4-07fa8e-kserve-self-signed-certs\"" Oct 30 21:35:51.286170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.286137 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-home\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.286170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.286177 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-model-cache\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.286401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.286198 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-tmp-dir\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.286438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.286398 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-dshm\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.286472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.286448 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6ls\" (UniqueName: \"kubernetes.io/projected/9e930fb0-4c95-4e49-aba1-2db4356980df-kube-api-access-hp6ls\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.286544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.286530 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e930fb0-4c95-4e49-aba1-2db4356980df-tls-certs\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.388078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.387981 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-dshm\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.388078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.388044 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6ls\" (UniqueName: \"kubernetes.io/projected/9e930fb0-4c95-4e49-aba1-2db4356980df-kube-api-access-hp6ls\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.388330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.388086 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e930fb0-4c95-4e49-aba1-2db4356980df-tls-certs\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.388330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.388174 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-home\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.388330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.388203 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-model-cache\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.388537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.388370 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-tmp-dir\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.388664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.388619 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-model-cache\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.388664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.388653 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-home\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.389012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.388716 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-tmp-dir\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.390583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.390562 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-dshm\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.390819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.390802 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e930fb0-4c95-4e49-aba1-2db4356980df-tls-certs\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.444853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.444824 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6ls\" (UniqueName: \"kubernetes.io/projected/9e930fb0-4c95-4e49-aba1-2db4356980df-kube-api-access-hp6ls\") pod \"distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.449631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.449602 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:35:51.844234 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:51.844204 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp"] Oct 30 21:35:51.845147 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:35:51.845115 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e930fb0_4c95_4e49_aba1_2db4356980df.slice/crio-b197aa5adbdcb04f3487f0350d9abf7cec628ac7922e0fe85a80a556eb036291 WatchSource:0}: Error finding container b197aa5adbdcb04f3487f0350d9abf7cec628ac7922e0fe85a80a556eb036291: Status 404 returned error can't find the container with id b197aa5adbdcb04f3487f0350d9abf7cec628ac7922e0fe85a80a556eb036291 Oct 30 21:35:52.847556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:52.847520 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" event={"ID":"9e930fb0-4c95-4e49-aba1-2db4356980df","Type":"ContainerStarted","Data":"42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da"} Oct 30 21:35:52.847963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:52.847567 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" event={"ID":"9e930fb0-4c95-4e49-aba1-2db4356980df","Type":"ContainerStarted","Data":"b197aa5adbdcb04f3487f0350d9abf7cec628ac7922e0fe85a80a556eb036291"} Oct 30 21:35:52.962174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:52.962122 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podStartSLOduration=1.9621085040000001 podStartE2EDuration="1.962108504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:52.960210028 +0000 UTC m=+1143.093583226" watchObservedRunningTime="2026-10-30 21:35:52.962108504 +0000 UTC m=+1143.095481764" Oct 30 21:35:53.450142 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:35:53.450108 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:36:01.450577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:36:01.450543 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:36:01.464195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:36:01.464170 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:36:01.927242 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:36:01.927207 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:36:12.479130 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:36:12.479089 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:36:42.477996 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:36:42.477959 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:37:03.334948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:03.334915 4526 generic.go:356] "Generic (PLEG): container finished" podID="e46d54e3-d4cc-4413-8fb1-0610b62b1b4c" containerID="16adf9a62e7e47946363907cb9bcd38131faf0874f8dafd8fa2d291e79438e29" exitCode=0 Oct 30 21:37:03.335362 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:03.334989 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9lshl-w29xw" event={"ID":"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c","Type":"ContainerDied","Data":"16adf9a62e7e47946363907cb9bcd38131faf0874f8dafd8fa2d291e79438e29"} Oct 30 21:37:04.497660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.497629 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:37:04.571579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.571546 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-archives-path\") pod \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " Oct 30 21:37:04.571753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.571668 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-trusted-ca-bundle\") pod \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " Oct 30 21:37:04.571753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.571691 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-kube-api-access-dcq8g\" (UniqueName: \"kubernetes.io/projected/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-kube-api-access-dcq8g\") pod \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " Oct 30 21:37:04.571831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.571818 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-service-ca-bundle\") pod \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\" (UID: \"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c\") " Oct 30 21:37:04.571886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.571854 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-archives-path" pod "e46d54e3-d4cc-4413-8fb1-0610b62b1b4c" (UID: "e46d54e3-d4cc-4413-8fb1-0610b62b1b4c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:04.572091 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.572067 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-trusted-ca-bundle" pod "e46d54e3-d4cc-4413-8fb1-0610b62b1b4c" (UID: "e46d54e3-d4cc-4413-8fb1-0610b62b1b4c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:04.572177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.572128 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-service-ca-bundle" pod "e46d54e3-d4cc-4413-8fb1-0610b62b1b4c" (UID: "e46d54e3-d4cc-4413-8fb1-0610b62b1b4c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:04.572332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.572313 4526 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-archives-path\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:04.572389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.572339 4526 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-trusted-ca-bundle\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:04.572389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.572355 4526 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-service-ca-bundle\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:04.574069 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.574044 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-kube-api-access-dcq8g" pod "e46d54e3-d4cc-4413-8fb1-0610b62b1b4c" (UID: "e46d54e3-d4cc-4413-8fb1-0610b62b1b4c"). InnerVolumeSpecName "kube-api-access-dcq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:04.674127 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:04.674028 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcq8g\" (UniqueName: \"kubernetes.io/projected/e46d54e3-d4cc-4413-8fb1-0610b62b1b4c-kube-api-access-dcq8g\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:05.348601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:05.348574 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9lshl-w29xw" Oct 30 21:37:05.348786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:05.348611 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9lshl-w29xw" event={"ID":"e46d54e3-d4cc-4413-8fb1-0610b62b1b4c","Type":"ContainerDied","Data":"c8eaf2ae051c58082bd75ec6e5d861527c2baae18244d290cf022685a907b6f1"} Oct 30 21:37:05.348786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:05.348665 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8eaf2ae051c58082bd75ec6e5d861527c2baae18244d290cf022685a907b6f1" Oct 30 21:37:08.548826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.548783 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt"] Oct 30 21:37:08.549538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.549519 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e46d54e3-d4cc-4413-8fb1-0610b62b1b4c" containerName="insights-gathering" Oct 30 21:37:08.549639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.549539 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46d54e3-d4cc-4413-8fb1-0610b62b1b4c" containerName="insights-gathering" Oct 30 21:37:08.549712 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.549698 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46d54e3-d4cc-4413-8fb1-0610b62b1b4c" containerName="insights-gathering" Oct 30 21:37:08.553020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.552998 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.606196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.606163 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:37:08.606196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.606173 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:08.655048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.655015 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:37:08.655048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.655013 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:37:08.658795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.658776 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt"] Oct 30 21:37:08.716371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.716342 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5hs\" (UniqueName: \"kubernetes.io/projected/db135340-6f8f-4de0-adfa-a463ef6e6db4-kube-api-access-vr5hs\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.716566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.716392 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.716566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.716477 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.818110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.818005 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5hs\" (UniqueName: \"kubernetes.io/projected/db135340-6f8f-4de0-adfa-a463ef6e6db4-kube-api-access-vr5hs\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.818110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.818061 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.818110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.818100 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.818713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.818695 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.820676 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.820658 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:08.874886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:08.874854 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5hs\" (UniqueName: \"kubernetes.io/projected/db135340-6f8f-4de0-adfa-a463ef6e6db4-kube-api-access-vr5hs\") pod \"maas-api-revoke-sub-b26da4e2-rwtvt\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:09.166811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:09.166725 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:09.546028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:09.545993 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt"] Oct 30 21:37:09.546905 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:37:09.546881 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb135340_6f8f_4de0_adfa_a463ef6e6db4.slice/crio-53888bf21fbcb6bd35bbb641aa694d5bb9bb31c2506d259ec807ecb0b3d6f61f WatchSource:0}: Error finding container 53888bf21fbcb6bd35bbb641aa694d5bb9bb31c2506d259ec807ecb0b3d6f61f: Status 404 returned error can't find the container with id 53888bf21fbcb6bd35bbb641aa694d5bb9bb31c2506d259ec807ecb0b3d6f61f Oct 30 21:37:10.385862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:10.385832 4526 generic.go:356] "Generic (PLEG): container finished" podID="db135340-6f8f-4de0-adfa-a463ef6e6db4" containerID="debb5ba68ac3f52e100fe054a5e857c473e700c8be10c5b979b9e2633bfc13b1" exitCode=0 Oct 30 21:37:10.386375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:10.385903 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" event={"ID":"db135340-6f8f-4de0-adfa-a463ef6e6db4","Type":"ContainerDied","Data":"debb5ba68ac3f52e100fe054a5e857c473e700c8be10c5b979b9e2633bfc13b1"} Oct 30 21:37:10.386375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:10.385966 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" event={"ID":"db135340-6f8f-4de0-adfa-a463ef6e6db4","Type":"ContainerStarted","Data":"53888bf21fbcb6bd35bbb641aa694d5bb9bb31c2506d259ec807ecb0b3d6f61f"} Oct 30 21:37:11.535763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.535732 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:11.655860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.655777 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db135340-6f8f-4de0-adfa-a463ef6e6db4-kube-api-access-vr5hs\" (UniqueName: \"kubernetes.io/projected/db135340-6f8f-4de0-adfa-a463ef6e6db4-kube-api-access-vr5hs\") pod \"db135340-6f8f-4de0-adfa-a463ef6e6db4\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " Oct 30 21:37:11.656045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.655910 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-openshift-ca\") pod \"db135340-6f8f-4de0-adfa-a463ef6e6db4\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " Oct 30 21:37:11.656137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.656069 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-xks-ca\") pod \"db135340-6f8f-4de0-adfa-a463ef6e6db4\" (UID: \"db135340-6f8f-4de0-adfa-a463ef6e6db4\") " Oct 30 21:37:11.656316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.656291 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-openshift-ca" pod "db135340-6f8f-4de0-adfa-a463ef6e6db4" (UID: "db135340-6f8f-4de0-adfa-a463ef6e6db4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:11.656571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.656547 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:11.658376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.658355 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db135340-6f8f-4de0-adfa-a463ef6e6db4-kube-api-access-vr5hs" pod "db135340-6f8f-4de0-adfa-a463ef6e6db4" (UID: "db135340-6f8f-4de0-adfa-a463ef6e6db4"). InnerVolumeSpecName "kube-api-access-vr5hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:11.658478 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.658412 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-xks-ca" pod "db135340-6f8f-4de0-adfa-a463ef6e6db4" (UID: "db135340-6f8f-4de0-adfa-a463ef6e6db4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:11.757921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.757884 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db135340-6f8f-4de0-adfa-a463ef6e6db4-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:11.757921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:11.757914 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5hs\" (UniqueName: \"kubernetes.io/projected/db135340-6f8f-4de0-adfa-a463ef6e6db4-kube-api-access-vr5hs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:12.134978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.134944 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m"] Oct 30 21:37:12.135473 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.135459 4526 cpu_manager.go:414] "containerMap: removing container" podUID="db135340-6f8f-4de0-adfa-a463ef6e6db4" containerName="revoke-keys" Oct 30 21:37:12.135552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.135475 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db135340-6f8f-4de0-adfa-a463ef6e6db4" containerName="revoke-keys" Oct 30 21:37:12.135607 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.135597 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="db135340-6f8f-4de0-adfa-a463ef6e6db4" containerName="revoke-keys" Oct 30 21:37:12.139014 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.138989 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.192509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.192419 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"distinct-2-w2-446ca2-kserve-self-signed-certs\"" Oct 30 21:37:12.244664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.244571 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m"] Oct 30 21:37:12.263986 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.263961 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tmp-dir\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.264181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.263995 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tls-certs\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.264181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.264025 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-dshm\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.264297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.264199 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-home\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.264297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.264253 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-model-cache\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.264366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.264333 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr9k6\" (UniqueName: \"kubernetes.io/projected/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-kube-api-access-pr9k6\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.365938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.365901 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tmp-dir\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.365938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.365942 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tls-certs\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.366192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.365967 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-dshm\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.366192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.366026 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-home\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.366192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.366064 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-model-cache\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.366192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.366102 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr9k6\" (UniqueName: \"kubernetes.io/projected/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-kube-api-access-pr9k6\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.366392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.366370 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tmp-dir\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.366392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.366381 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-home\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.366700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.366678 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-model-cache\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.369034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.369012 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-dshm\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.369863 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.369837 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tls-certs\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.399384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.399355 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" Oct 30 21:37:12.399535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.399390 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt" event={"ID":"db135340-6f8f-4de0-adfa-a463ef6e6db4","Type":"ContainerDied","Data":"53888bf21fbcb6bd35bbb641aa694d5bb9bb31c2506d259ec807ecb0b3d6f61f"} Oct 30 21:37:12.399535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.399440 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53888bf21fbcb6bd35bbb641aa694d5bb9bb31c2506d259ec807ecb0b3d6f61f" Oct 30 21:37:12.422902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.422865 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr9k6\" (UniqueName: \"kubernetes.io/projected/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-kube-api-access-pr9k6\") pod \"distinct-2-w2-446ca2-kserve-7499f65988-s954m\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.451287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.451264 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:12.479002 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:37:12.478962 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:37:12.833914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:12.833875 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m"] Oct 30 21:37:12.834330 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:37:12.834133 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd88f814_b3c3_48f9_a4ee_e9c7bafb92b0.slice/crio-efff3d94cefbac178494d9bf47ee2c971add39c24063563bef1afd91b46e805e WatchSource:0}: Error finding container efff3d94cefbac178494d9bf47ee2c971add39c24063563bef1afd91b46e805e: Status 404 returned error can't find the container with id efff3d94cefbac178494d9bf47ee2c971add39c24063563bef1afd91b46e805e Oct 30 21:37:13.407217 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:13.407190 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" event={"ID":"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0","Type":"ContainerStarted","Data":"4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8"} Oct 30 21:37:13.407400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:13.407228 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" event={"ID":"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0","Type":"ContainerStarted","Data":"efff3d94cefbac178494d9bf47ee2c971add39c24063563bef1afd91b46e805e"} Oct 30 21:37:13.451512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:13.451458 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:13.521375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:13.521326 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podStartSLOduration=1.5213107529999998 podStartE2EDuration="1.521310753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:13.519522634 +0000 UTC m=+1223.652895829" watchObservedRunningTime="2026-10-30 21:37:13.521310753 +0000 UTC m=+1223.654683950" Oct 30 21:37:20.055336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:20.055297 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf"] Oct 30 21:37:20.102682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:20.102647 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10d2545-6p5bf"] Oct 30 21:37:21.046474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:21.046435 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n"] Oct 30 21:37:21.095401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:21.095366 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a8b2fe-bdx9n"] Oct 30 21:37:21.986150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:21.986119 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347013c5-51f5-46de-8b5e-3caf3c2848e6" path="/var/lib/kubelet/pods/347013c5-51f5-46de-8b5e-3caf3c2848e6/volumes" Oct 30 21:37:21.986586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:21.986573 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8864de-322e-4f94-84a1-9857a8ae5946" path="/var/lib/kubelet/pods/fb8864de-322e-4f94-84a1-9857a8ae5946/volumes" Oct 30 21:37:22.451989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:22.451954 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:22.465054 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:22.465025 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:22.484184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:22.484158 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:37:22.994701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:22.994668 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs"] Oct 30 21:37:23.000088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.000067 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.052054 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.052026 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:37:23.052324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.052042 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:23.054985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.054962 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs"] Oct 30 21:37:23.079613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.079578 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76bh\" (UniqueName: \"kubernetes.io/projected/d793c46d-16e0-4229-896f-0cf382f07deb-kube-api-access-s76bh\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.079779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.079641 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.079779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.079764 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.099395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.099361 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:37:23.099605 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.099590 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:37:23.181570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.181540 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.181763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.181594 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.181763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.181730 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76bh\" (UniqueName: \"kubernetes.io/projected/d793c46d-16e0-4229-896f-0cf382f07deb-kube-api-access-s76bh\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.182288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.182215 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.184140 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.184118 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.237924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.237891 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76bh\" (UniqueName: \"kubernetes.io/projected/d793c46d-16e0-4229-896f-0cf382f07deb-kube-api-access-s76bh\") pod \"maas-api-revoke-sub-f22b6a5f-h8qzs\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.313160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.313126 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:23.703290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:23.702883 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs"] Oct 30 21:37:24.487111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:24.487076 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" event={"ID":"d793c46d-16e0-4229-896f-0cf382f07deb","Type":"ContainerStarted","Data":"abe0230fabbebc2afe6f7c2248a04770e3c8dc28c6b9d7384189106886d33c17"} Oct 30 21:37:24.487289 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:24.487125 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" event={"ID":"d793c46d-16e0-4229-896f-0cf382f07deb","Type":"ContainerStarted","Data":"f869fc4a994a7383b97c32353ecf8f09f37d86f9dc17b35256aa9928e28d1246"} Oct 30 21:37:24.601732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:24.601683 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" podStartSLOduration=2.601668728 podStartE2EDuration="2.601668728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:24.599771779 +0000 UTC m=+1234.733144979" watchObservedRunningTime="2026-10-30 21:37:24.601668728 +0000 UTC m=+1234.735041963" Oct 30 21:37:25.495804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:25.495767 4526 generic.go:356] "Generic (PLEG): container finished" podID="d793c46d-16e0-4229-896f-0cf382f07deb" containerID="abe0230fabbebc2afe6f7c2248a04770e3c8dc28c6b9d7384189106886d33c17" exitCode=0 Oct 30 21:37:25.496227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:25.495823 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" event={"ID":"d793c46d-16e0-4229-896f-0cf382f07deb","Type":"ContainerDied","Data":"abe0230fabbebc2afe6f7c2248a04770e3c8dc28c6b9d7384189106886d33c17"} Oct 30 21:37:26.664137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.664113 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:26.722094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.722056 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d793c46d-16e0-4229-896f-0cf382f07deb-kube-api-access-s76bh\" (UniqueName: \"kubernetes.io/projected/d793c46d-16e0-4229-896f-0cf382f07deb-kube-api-access-s76bh\") pod \"d793c46d-16e0-4229-896f-0cf382f07deb\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " Oct 30 21:37:26.722094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.722097 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-openshift-ca\") pod \"d793c46d-16e0-4229-896f-0cf382f07deb\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " Oct 30 21:37:26.722313 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.722212 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-xks-ca\") pod \"d793c46d-16e0-4229-896f-0cf382f07deb\" (UID: \"d793c46d-16e0-4229-896f-0cf382f07deb\") " Oct 30 21:37:26.722475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.722451 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-openshift-ca" pod "d793c46d-16e0-4229-896f-0cf382f07deb" (UID: "d793c46d-16e0-4229-896f-0cf382f07deb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:26.722717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.722700 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:26.724450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.724426 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-xks-ca" pod "d793c46d-16e0-4229-896f-0cf382f07deb" (UID: "d793c46d-16e0-4229-896f-0cf382f07deb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:26.724575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.724454 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d793c46d-16e0-4229-896f-0cf382f07deb-kube-api-access-s76bh" pod "d793c46d-16e0-4229-896f-0cf382f07deb" (UID: "d793c46d-16e0-4229-896f-0cf382f07deb"). InnerVolumeSpecName "kube-api-access-s76bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:26.823984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.823953 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s76bh\" (UniqueName: \"kubernetes.io/projected/d793c46d-16e0-4229-896f-0cf382f07deb-kube-api-access-s76bh\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:26.823984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:26.823981 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d793c46d-16e0-4229-896f-0cf382f07deb-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:37:27.510600 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:27.510562 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" event={"ID":"d793c46d-16e0-4229-896f-0cf382f07deb","Type":"ContainerDied","Data":"f869fc4a994a7383b97c32353ecf8f09f37d86f9dc17b35256aa9928e28d1246"} Oct 30 21:37:27.510805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:27.510625 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f869fc4a994a7383b97c32353ecf8f09f37d86f9dc17b35256aa9928e28d1246" Oct 30 21:37:27.510805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:27.510575 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs" Oct 30 21:37:38.165201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.165164 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw"] Oct 30 21:37:38.165739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.165724 4526 cpu_manager.go:414] "containerMap: removing container" podUID="d793c46d-16e0-4229-896f-0cf382f07deb" containerName="revoke-keys" Oct 30 21:37:38.165739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.165740 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d793c46d-16e0-4229-896f-0cf382f07deb" containerName="revoke-keys" Oct 30 21:37:38.165831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.165822 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="d793c46d-16e0-4229-896f-0cf382f07deb" containerName="revoke-keys" Oct 30 21:37:38.169218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.169202 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.256619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.256585 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.256817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.256654 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f07d391-d13a-482e-a61e-d7689bf80c30-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.256817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.256753 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.256817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.256796 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.256979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.256916 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.256979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.256941 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9j9w\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-kube-api-access-g9j9w\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.256979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.256968 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.257128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.257090 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.257181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.257164 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.274412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.274382 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw"] Oct 30 21:37:38.359019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.358981 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359060 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f07d391-d13a-482e-a61e-d7689bf80c30-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359120 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359172 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359251 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359279 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9j9w\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-kube-api-access-g9j9w\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359335 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359631 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359668 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359857 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.359969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.359950 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f07d391-d13a-482e-a61e-d7689bf80c30-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.360039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.360017 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.360107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.360090 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.360374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.360355 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.361850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.361829 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.362629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.362598 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.416022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.415942 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.416022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.416010 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9j9w\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-kube-api-access-g9j9w\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.482618 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.482582 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:38.916116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.916080 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw"] Oct 30 21:37:38.955112 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:37:38.955087 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f07d391_d13a_482e_a61e_d7689bf80c30.slice/crio-a28f64bbfe39cce1b603dc5a9a32ca418aaa0a556366732bbbbbfe66e0334b4d WatchSource:0}: Error finding container a28f64bbfe39cce1b603dc5a9a32ca418aaa0a556366732bbbbbfe66e0334b4d: Status 404 returned error can't find the container with id a28f64bbfe39cce1b603dc5a9a32ca418aaa0a556366732bbbbbfe66e0334b4d Oct 30 21:37:38.957411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.957382 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:37:38.957465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.957451 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:37:38.957512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:38.957479 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:37:39.593264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:39.593231 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" event={"ID":"3f07d391-d13a-482e-a61e-d7689bf80c30","Type":"ContainerStarted","Data":"9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21"} Oct 30 21:37:39.593741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:39.593284 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" event={"ID":"3f07d391-d13a-482e-a61e-d7689bf80c30","Type":"ContainerStarted","Data":"a28f64bbfe39cce1b603dc5a9a32ca418aaa0a556366732bbbbbfe66e0334b4d"} Oct 30 21:37:40.483027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:40.482998 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:40.484849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:40.484815 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:37:40.484997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:40.484874 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:37:41.483822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:41.483789 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:37:41.484224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:41.483850 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:37:42.479480 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:37:42.479444 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:37:42.483714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:42.483683 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:37:42.483845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:42.483757 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:37:43.482989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:43.482950 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:37:43.483179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:43.483005 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:37:44.483365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:44.483325 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:37:44.483956 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:44.483396 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:37:45.046583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:45.046531 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podStartSLOduration=7.04651241 podStartE2EDuration="7.04651241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:39.823697812 +0000 UTC m=+1249.957071011" watchObservedRunningTime="2026-10-30 21:37:45.04651241 +0000 UTC m=+1255.179885608" Oct 30 21:37:45.047962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:45.047943 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x"] Oct 30 21:37:45.097441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:45.097401 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5dc42e-mwq6x"] Oct 30 21:37:45.483516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:45.483422 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:37:45.483895 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:45.483547 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 21:37:45.986341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:45.986299 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b967e8-1484-48c9-b52a-f8e604df8576" path="/var/lib/kubelet/pods/c9b967e8-1484-48c9-b52a-f8e604df8576/volumes" Oct 30 21:37:46.488163 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:46.488134 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:46.488680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:46.488661 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:46.489327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:46.489308 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:37:55.050146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:55.050108 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7"] Oct 30 21:37:55.098940 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:55.098908 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86170231-kgvv7"] Oct 30 21:37:55.985719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:37:55.985689 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93671712-15ba-4b9a-a476-a97285a2268d" path="/var/lib/kubelet/pods/93671712-15ba-4b9a-a476-a97285a2268d/volumes" Oct 30 21:38:06.624383 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.624348 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w"] Oct 30 21:38:06.629937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.629914 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.684977 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.684950 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w"] Oct 30 21:38:06.723943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.723912 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ftk\" (UniqueName: \"kubernetes.io/projected/8357554a-60c8-4c16-b4da-331f30a57ee5-kube-api-access-v6ftk\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.724103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.723984 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.724103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.724075 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.730230 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.730206 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:06.731293 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.731273 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:06.731360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.731293 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:06.731360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.731303 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:38:06.825241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.825212 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.825446 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.825330 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ftk\" (UniqueName: \"kubernetes.io/projected/8357554a-60c8-4c16-b4da-331f30a57ee5-kube-api-access-v6ftk\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.825446 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.825386 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.826104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.826079 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.827794 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.827769 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.881763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.881688 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ftk\" (UniqueName: \"kubernetes.io/projected/8357554a-60c8-4c16-b4da-331f30a57ee5-kube-api-access-v6ftk\") pod \"maas-api-revoke-sub-b79a13d9-x7r9w\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:06.944142 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:06.944096 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:07.321731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:07.321698 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w"] Oct 30 21:38:07.322252 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:07.322228 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8357554a_60c8_4c16_b4da_331f30a57ee5.slice/crio-bf70c9c5d1604191d88f06bd6ffd70c1c7aa2ef4e6f54a5b8dbf0cf59a941e7b WatchSource:0}: Error finding container bf70c9c5d1604191d88f06bd6ffd70c1c7aa2ef4e6f54a5b8dbf0cf59a941e7b: Status 404 returned error can't find the container with id bf70c9c5d1604191d88f06bd6ffd70c1c7aa2ef4e6f54a5b8dbf0cf59a941e7b Oct 30 21:38:07.825181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:07.825143 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" event={"ID":"8357554a-60c8-4c16-b4da-331f30a57ee5","Type":"ContainerStarted","Data":"2ff20305195b717a32b7d45798fab29c518802e41304d1607bbf5c465c63fe50"} Oct 30 21:38:07.825677 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:07.825204 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" event={"ID":"8357554a-60c8-4c16-b4da-331f30a57ee5","Type":"ContainerStarted","Data":"bf70c9c5d1604191d88f06bd6ffd70c1c7aa2ef4e6f54a5b8dbf0cf59a941e7b"} Oct 30 21:38:07.940221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:07.940125 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" podStartSLOduration=1.940110652 podStartE2EDuration="1.940110652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:07.938567353 +0000 UTC m=+1278.071940552" watchObservedRunningTime="2026-10-30 21:38:07.940110652 +0000 UTC m=+1278.073483850" Oct 30 21:38:08.833681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:08.833647 4526 generic.go:356] "Generic (PLEG): container finished" podID="8357554a-60c8-4c16-b4da-331f30a57ee5" containerID="2ff20305195b717a32b7d45798fab29c518802e41304d1607bbf5c465c63fe50" exitCode=0 Oct 30 21:38:08.834125 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:08.833733 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" event={"ID":"8357554a-60c8-4c16-b4da-331f30a57ee5","Type":"ContainerDied","Data":"2ff20305195b717a32b7d45798fab29c518802e41304d1607bbf5c465c63fe50"} Oct 30 21:38:10.000223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.000196 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:10.166192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.166088 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-openshift-ca\") pod \"8357554a-60c8-4c16-b4da-331f30a57ee5\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " Oct 30 21:38:10.166383 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.166203 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-xks-ca\") pod \"8357554a-60c8-4c16-b4da-331f30a57ee5\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " Oct 30 21:38:10.166383 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.166257 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8357554a-60c8-4c16-b4da-331f30a57ee5-kube-api-access-v6ftk\" (UniqueName: \"kubernetes.io/projected/8357554a-60c8-4c16-b4da-331f30a57ee5-kube-api-access-v6ftk\") pod \"8357554a-60c8-4c16-b4da-331f30a57ee5\" (UID: \"8357554a-60c8-4c16-b4da-331f30a57ee5\") " Oct 30 21:38:10.166555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.166515 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-openshift-ca" pod "8357554a-60c8-4c16-b4da-331f30a57ee5" (UID: "8357554a-60c8-4c16-b4da-331f30a57ee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:10.166827 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.166800 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:10.168663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.168633 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8357554a-60c8-4c16-b4da-331f30a57ee5-kube-api-access-v6ftk" pod "8357554a-60c8-4c16-b4da-331f30a57ee5" (UID: "8357554a-60c8-4c16-b4da-331f30a57ee5"). InnerVolumeSpecName "kube-api-access-v6ftk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:10.168663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.168641 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-xks-ca" pod "8357554a-60c8-4c16-b4da-331f30a57ee5" (UID: "8357554a-60c8-4c16-b4da-331f30a57ee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:10.268499 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.268459 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8357554a-60c8-4c16-b4da-331f30a57ee5-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:10.268692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.268514 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ftk\" (UniqueName: \"kubernetes.io/projected/8357554a-60c8-4c16-b4da-331f30a57ee5-kube-api-access-v6ftk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:10.849496 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.849466 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" Oct 30 21:38:10.849688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.849520 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w" event={"ID":"8357554a-60c8-4c16-b4da-331f30a57ee5","Type":"ContainerDied","Data":"bf70c9c5d1604191d88f06bd6ffd70c1c7aa2ef4e6f54a5b8dbf0cf59a941e7b"} Oct 30 21:38:10.849688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:10.849571 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf70c9c5d1604191d88f06bd6ffd70c1c7aa2ef4e6f54a5b8dbf0cf59a941e7b" Oct 30 21:38:12.477713 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:38:12.477681 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:38:13.053586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:13.053552 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp"] Oct 30 21:38:13.097939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:13.097900 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a2209-59rhp"] Oct 30 21:38:13.985861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:13.985830 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8251fafa-ad43-4470-b1ba-6f8849ccc8ff" path="/var/lib/kubelet/pods/8251fafa-ad43-4470-b1ba-6f8849ccc8ff/volumes" Oct 30 21:38:15.045920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:15.045887 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh"] Oct 30 21:38:15.095322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:15.095287 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460290b5-gk5xh"] Oct 30 21:38:15.986596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:15.986568 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99fc9fcb-2145-4938-a926-ab6742153162" path="/var/lib/kubelet/pods/99fc9fcb-2145-4938-a926-ab6742153162/volumes" Oct 30 21:38:16.696561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.696528 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq"] Oct 30 21:38:16.697080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.697068 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8357554a-60c8-4c16-b4da-331f30a57ee5" containerName="revoke-keys" Oct 30 21:38:16.697181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.697082 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8357554a-60c8-4c16-b4da-331f30a57ee5" containerName="revoke-keys" Oct 30 21:38:16.697181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.697167 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8357554a-60c8-4c16-b4da-331f30a57ee5" containerName="revoke-keys" Oct 30 21:38:16.701877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.701861 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.744401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.744362 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlwzk\" (UniqueName: \"kubernetes.io/projected/13f01b59-20f2-4e0f-a3bd-41807c097d88-kube-api-access-rlwzk\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.744596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.744446 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.744691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.744608 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.754171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.754142 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:16.754171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.754151 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:16.754337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.754176 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:16.803455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.803424 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:38:16.809629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.809599 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq"] Oct 30 21:38:16.846237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.845607 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.846237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.845722 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.846237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.845895 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlwzk\" (UniqueName: \"kubernetes.io/projected/13f01b59-20f2-4e0f-a3bd-41807c097d88-kube-api-access-rlwzk\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.846957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.846934 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.848639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.848610 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:16.902284 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:16.902253 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlwzk\" (UniqueName: \"kubernetes.io/projected/13f01b59-20f2-4e0f-a3bd-41807c097d88-kube-api-access-rlwzk\") pod \"maas-api-revoke-sub-93f76d0e-fn6bq\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:17.015428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:17.015348 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:17.395933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:17.395902 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq"] Oct 30 21:38:17.396744 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:17.396721 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f01b59_20f2_4e0f_a3bd_41807c097d88.slice/crio-944f4f6022e8beff4396cbba29246525213f66d0b8ea68df134d72dddd710760 WatchSource:0}: Error finding container 944f4f6022e8beff4396cbba29246525213f66d0b8ea68df134d72dddd710760: Status 404 returned error can't find the container with id 944f4f6022e8beff4396cbba29246525213f66d0b8ea68df134d72dddd710760 Oct 30 21:38:17.900055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:17.900024 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" event={"ID":"13f01b59-20f2-4e0f-a3bd-41807c097d88","Type":"ContainerStarted","Data":"f1ff63d614f341fc1d08927a86bacbb1fa39b38ceb0e4cef88ee0a519f301349"} Oct 30 21:38:17.900509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:17.900078 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" event={"ID":"13f01b59-20f2-4e0f-a3bd-41807c097d88","Type":"ContainerStarted","Data":"944f4f6022e8beff4396cbba29246525213f66d0b8ea68df134d72dddd710760"} Oct 30 21:38:18.020689 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:18.020583 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" podStartSLOduration=2.020568644 podStartE2EDuration="2.020568644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:18.016883488 +0000 UTC m=+1288.150256686" watchObservedRunningTime="2026-10-30 21:38:18.020568644 +0000 UTC m=+1288.153941841" Oct 30 21:38:18.909839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:18.909533 4526 generic.go:356] "Generic (PLEG): container finished" podID="13f01b59-20f2-4e0f-a3bd-41807c097d88" containerID="f1ff63d614f341fc1d08927a86bacbb1fa39b38ceb0e4cef88ee0a519f301349" exitCode=0 Oct 30 21:38:18.909839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:18.909778 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" event={"ID":"13f01b59-20f2-4e0f-a3bd-41807c097d88","Type":"ContainerDied","Data":"f1ff63d614f341fc1d08927a86bacbb1fa39b38ceb0e4cef88ee0a519f301349"} Oct 30 21:38:20.091156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.091131 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:20.184034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.183945 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f01b59-20f2-4e0f-a3bd-41807c097d88-kube-api-access-rlwzk\" (UniqueName: \"kubernetes.io/projected/13f01b59-20f2-4e0f-a3bd-41807c097d88-kube-api-access-rlwzk\") pod \"13f01b59-20f2-4e0f-a3bd-41807c097d88\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " Oct 30 21:38:20.184221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.184033 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-xks-ca\") pod \"13f01b59-20f2-4e0f-a3bd-41807c097d88\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " Oct 30 21:38:20.184221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.184203 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-openshift-ca\") pod \"13f01b59-20f2-4e0f-a3bd-41807c097d88\" (UID: \"13f01b59-20f2-4e0f-a3bd-41807c097d88\") " Oct 30 21:38:20.184986 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.184941 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-openshift-ca" pod "13f01b59-20f2-4e0f-a3bd-41807c097d88" (UID: "13f01b59-20f2-4e0f-a3bd-41807c097d88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:20.186029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.186003 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:20.195068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.195037 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f01b59-20f2-4e0f-a3bd-41807c097d88-kube-api-access-rlwzk" pod "13f01b59-20f2-4e0f-a3bd-41807c097d88" (UID: "13f01b59-20f2-4e0f-a3bd-41807c097d88"). InnerVolumeSpecName "kube-api-access-rlwzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:20.196825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.196799 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-xks-ca" pod "13f01b59-20f2-4e0f-a3bd-41807c097d88" (UID: "13f01b59-20f2-4e0f-a3bd-41807c097d88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:20.286948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.286907 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlwzk\" (UniqueName: \"kubernetes.io/projected/13f01b59-20f2-4e0f-a3bd-41807c097d88-kube-api-access-rlwzk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:20.286948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.286945 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f01b59-20f2-4e0f-a3bd-41807c097d88-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:20.927411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.927376 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" Oct 30 21:38:20.927411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.927382 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq" event={"ID":"13f01b59-20f2-4e0f-a3bd-41807c097d88","Type":"ContainerDied","Data":"944f4f6022e8beff4396cbba29246525213f66d0b8ea68df134d72dddd710760"} Oct 30 21:38:20.927692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:20.927456 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944f4f6022e8beff4396cbba29246525213f66d0b8ea68df134d72dddd710760" Oct 30 21:38:22.818063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.818027 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp"] Oct 30 21:38:22.818585 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.818572 4526 cpu_manager.go:414] "containerMap: removing container" podUID="13f01b59-20f2-4e0f-a3bd-41807c097d88" containerName="revoke-keys" Oct 30 21:38:22.818635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.818587 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f01b59-20f2-4e0f-a3bd-41807c097d88" containerName="revoke-keys" Oct 30 21:38:22.818704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.818696 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f01b59-20f2-4e0f-a3bd-41807c097d88" containerName="revoke-keys" Oct 30 21:38:22.822246 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.822224 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:22.878234 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.878201 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp"] Oct 30 21:38:22.916725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.916694 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:22.916908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.916885 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkqf\" (UniqueName: \"kubernetes.io/projected/4cb00eca-9882-492d-bc92-b26bc53fa01e-kube-api-access-pfkqf\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:22.917022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.917005 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:22.923221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.923195 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:22.923221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.923217 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:22.923397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.923202 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:22.923397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.923275 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:38:22.929310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.929288 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4"] Oct 30 21:38:22.933392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.933378 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:22.993434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:22.993406 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4"] Oct 30 21:38:23.017710 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.017676 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.017710 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.017711 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:23.017972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.017781 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.017972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.017825 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.017972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.017889 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.017972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.017914 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.018191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.018081 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkqf\" (UniqueName: \"kubernetes.io/projected/4cb00eca-9882-492d-bc92-b26bc53fa01e-kube-api-access-pfkqf\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:23.018191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.018119 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.018299 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.018258 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:23.018356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.018312 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.018415 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.018372 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.018470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.018415 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52sb8\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-kube-api-access-52sb8\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.018993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.018958 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:23.020751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.020729 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:23.074897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.074818 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkqf\" (UniqueName: \"kubernetes.io/projected/4cb00eca-9882-492d-bc92-b26bc53fa01e-kube-api-access-pfkqf\") pod \"maas-api-revoke-sub-f24db36e-mgcdp\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:23.119348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119316 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119363 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119401 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119417 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119447 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119540 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119578 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119620 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52sb8\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-kube-api-access-52sb8\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119688 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119809 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.119916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.119852 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.120109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.120010 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.120148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.120125 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.120224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.120205 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.121959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.121938 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.122167 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.122151 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.135433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.135408 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:23.175665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.175630 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.175804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.175717 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52sb8\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-kube-api-access-52sb8\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.247652 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.247621 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:23.510196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.510168 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp"] Oct 30 21:38:23.510995 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:23.510972 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb00eca_9882_492d_bc92_b26bc53fa01e.slice/crio-20b2979fbb7e137d4046a9e9c318aee6463b764f62bc63bb321db804fd17f569 WatchSource:0}: Error finding container 20b2979fbb7e137d4046a9e9c318aee6463b764f62bc63bb321db804fd17f569: Status 404 returned error can't find the container with id 20b2979fbb7e137d4046a9e9c318aee6463b764f62bc63bb321db804fd17f569 Oct 30 21:38:23.626731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.626689 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4"] Oct 30 21:38:23.627596 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:23.627569 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ee9708_4119_4fcd_b9b2_76943d9d3c8b.slice/crio-c6426ebc91f942cdea6728f91e5b15af6ca16eed6b67718a88c8946983babcd1 WatchSource:0}: Error finding container c6426ebc91f942cdea6728f91e5b15af6ca16eed6b67718a88c8946983babcd1: Status 404 returned error can't find the container with id c6426ebc91f942cdea6728f91e5b15af6ca16eed6b67718a88c8946983babcd1 Oct 30 21:38:23.630158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.630129 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:23.630228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.630194 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:23.630228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.630223 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:23.950034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.949994 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" event={"ID":"20ee9708-4119-4fcd-b9b2-76943d9d3c8b","Type":"ContainerStarted","Data":"45802f25775ee604396d4c925d78ade4b44befbb260ae9e0a871e6282568e920"} Oct 30 21:38:23.950561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.950051 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" event={"ID":"20ee9708-4119-4fcd-b9b2-76943d9d3c8b","Type":"ContainerStarted","Data":"c6426ebc91f942cdea6728f91e5b15af6ca16eed6b67718a88c8946983babcd1"} Oct 30 21:38:23.951642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.951615 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" event={"ID":"4cb00eca-9882-492d-bc92-b26bc53fa01e","Type":"ContainerStarted","Data":"ec585ff982b8e41c634cbfc122baca3536ff3e1f4eafdae1378d614df43fe77c"} Oct 30 21:38:23.951771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:23.951660 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" event={"ID":"4cb00eca-9882-492d-bc92-b26bc53fa01e","Type":"ContainerStarted","Data":"20b2979fbb7e137d4046a9e9c318aee6463b764f62bc63bb321db804fd17f569"} Oct 30 21:38:24.067137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:24.067023 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podStartSLOduration=2.06700742 podStartE2EDuration="2.06700742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:24.066641874 +0000 UTC m=+1294.200015083" watchObservedRunningTime="2026-10-30 21:38:24.06700742 +0000 UTC m=+1294.200380618" Oct 30 21:38:24.178215 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:24.178154 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" podStartSLOduration=2.178137698 podStartE2EDuration="2.178137698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:24.17789012 +0000 UTC m=+1294.311263317" watchObservedRunningTime="2026-10-30 21:38:24.178137698 +0000 UTC m=+1294.311510893" Oct 30 21:38:24.248364 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:24.248337 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:24.249674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:24.249654 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:38:24.249764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:24.249697 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:38:24.960009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:24.959915 4526 generic.go:356] "Generic (PLEG): container finished" podID="4cb00eca-9882-492d-bc92-b26bc53fa01e" containerID="ec585ff982b8e41c634cbfc122baca3536ff3e1f4eafdae1378d614df43fe77c" exitCode=0 Oct 30 21:38:24.960528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:24.960002 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" event={"ID":"4cb00eca-9882-492d-bc92-b26bc53fa01e","Type":"ContainerDied","Data":"ec585ff982b8e41c634cbfc122baca3536ff3e1f4eafdae1378d614df43fe77c"} Oct 30 21:38:25.247998 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:25.247907 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:38:25.247998 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:25.247972 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:38:26.163968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.163942 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:26.247958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.247871 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:38:26.247958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.247925 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:38:26.262971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.262943 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb00eca-9882-492d-bc92-b26bc53fa01e-kube-api-access-pfkqf\" (UniqueName: \"kubernetes.io/projected/4cb00eca-9882-492d-bc92-b26bc53fa01e-kube-api-access-pfkqf\") pod \"4cb00eca-9882-492d-bc92-b26bc53fa01e\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " Oct 30 21:38:26.263145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.263073 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-openshift-ca\") pod \"4cb00eca-9882-492d-bc92-b26bc53fa01e\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " Oct 30 21:38:26.263270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.263251 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-xks-ca\") pod \"4cb00eca-9882-492d-bc92-b26bc53fa01e\" (UID: \"4cb00eca-9882-492d-bc92-b26bc53fa01e\") " Oct 30 21:38:26.263433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.263409 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-openshift-ca" pod "4cb00eca-9882-492d-bc92-b26bc53fa01e" (UID: "4cb00eca-9882-492d-bc92-b26bc53fa01e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:26.263717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.263689 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:26.265329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.265301 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb00eca-9882-492d-bc92-b26bc53fa01e-kube-api-access-pfkqf" pod "4cb00eca-9882-492d-bc92-b26bc53fa01e" (UID: "4cb00eca-9882-492d-bc92-b26bc53fa01e"). InnerVolumeSpecName "kube-api-access-pfkqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:26.265421 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.265376 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-xks-ca" pod "4cb00eca-9882-492d-bc92-b26bc53fa01e" (UID: "4cb00eca-9882-492d-bc92-b26bc53fa01e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:26.364627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.364569 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb00eca-9882-492d-bc92-b26bc53fa01e-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:26.364627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.364610 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfkqf\" (UniqueName: \"kubernetes.io/projected/4cb00eca-9882-492d-bc92-b26bc53fa01e-kube-api-access-pfkqf\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:26.976224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.976190 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" event={"ID":"4cb00eca-9882-492d-bc92-b26bc53fa01e","Type":"ContainerDied","Data":"20b2979fbb7e137d4046a9e9c318aee6463b764f62bc63bb321db804fd17f569"} Oct 30 21:38:26.976392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.976236 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp" Oct 30 21:38:26.976392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:26.976274 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b2979fbb7e137d4046a9e9c318aee6463b764f62bc63bb321db804fd17f569" Oct 30 21:38:27.045960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.045920 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5"] Oct 30 21:38:27.091375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.091340 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1f35186-9vfx5"] Oct 30 21:38:27.248456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.248374 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:38:27.248456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.248432 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:38:27.465043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.465006 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl"] Oct 30 21:38:27.465578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.465563 4526 cpu_manager.go:414] "containerMap: removing container" podUID="4cb00eca-9882-492d-bc92-b26bc53fa01e" containerName="revoke-keys" Oct 30 21:38:27.465632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.465580 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb00eca-9882-492d-bc92-b26bc53fa01e" containerName="revoke-keys" Oct 30 21:38:27.465682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.465674 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb00eca-9882-492d-bc92-b26bc53fa01e" containerName="revoke-keys" Oct 30 21:38:27.469089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.469068 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.520870 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.520761 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:27.521882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.521194 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:27.521882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.521217 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:38:27.521882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.521237 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:27.526683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.526652 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl"] Oct 30 21:38:27.577609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.577580 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxql\" (UniqueName: \"kubernetes.io/projected/98c9ec3b-435e-4e24-ae23-bf4da48820db-kube-api-access-kpxql\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.577788 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.577736 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.577830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.577799 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.679405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.679371 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.679612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.679451 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxql\" (UniqueName: \"kubernetes.io/projected/98c9ec3b-435e-4e24-ae23-bf4da48820db-kube-api-access-kpxql\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.679612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.679578 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.680113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.680094 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.682130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.682105 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.736114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.736081 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxql\" (UniqueName: \"kubernetes.io/projected/98c9ec3b-435e-4e24-ae23-bf4da48820db-kube-api-access-kpxql\") pod \"maas-api-revoke-sub-a0032967-pw4hl\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.783276 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.783192 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:27.986010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:27.985966 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71dd9c35-5a0d-42ea-a30f-633a2c95fe3c" path="/var/lib/kubelet/pods/71dd9c35-5a0d-42ea-a30f-633a2c95fe3c/volumes" Oct 30 21:38:28.048247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.048217 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng"] Oct 30 21:38:28.052717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.052688 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt"] Oct 30 21:38:28.055964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.055945 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc968c57-2p8ng"] Oct 30 21:38:28.120943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.120902 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee78e7d-b97lt"] Oct 30 21:38:28.190598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.190571 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl"] Oct 30 21:38:28.191292 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:28.191267 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c9ec3b_435e_4e24_ae23_bf4da48820db.slice/crio-ab04067e04ab3fca9f0f893619e1dcebbf9c9415c97cbbb78e73f7baeb910aaa WatchSource:0}: Error finding container ab04067e04ab3fca9f0f893619e1dcebbf9c9415c97cbbb78e73f7baeb910aaa: Status 404 returned error can't find the container with id ab04067e04ab3fca9f0f893619e1dcebbf9c9415c97cbbb78e73f7baeb910aaa Oct 30 21:38:28.247980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.247950 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:38:28.248155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.248009 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:38:28.992886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.992855 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" event={"ID":"98c9ec3b-435e-4e24-ae23-bf4da48820db","Type":"ContainerStarted","Data":"fbf865f476ecb037cfe7aa873ec36362f4ca4ed1526755718cfc57bda37522f3"} Oct 30 21:38:28.992886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:28.992901 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" event={"ID":"98c9ec3b-435e-4e24-ae23-bf4da48820db","Type":"ContainerStarted","Data":"ab04067e04ab3fca9f0f893619e1dcebbf9c9415c97cbbb78e73f7baeb910aaa"} Oct 30 21:38:29.248847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:29.248770 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:38:29.248847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:29.248822 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:38:29.995165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:29.995132 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c5f415-2ae4-419f-8b5d-9147a4a41204" path="/var/lib/kubelet/pods/a9c5f415-2ae4-419f-8b5d-9147a4a41204/volumes" Oct 30 21:38:29.995702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:29.995630 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36a4d63-9c35-4824-9cf1-77c96d325cac" path="/var/lib/kubelet/pods/f36a4d63-9c35-4824-9cf1-77c96d325cac/volumes" Oct 30 21:38:30.001317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:30.001293 4526 generic.go:356] "Generic (PLEG): container finished" podID="98c9ec3b-435e-4e24-ae23-bf4da48820db" containerID="fbf865f476ecb037cfe7aa873ec36362f4ca4ed1526755718cfc57bda37522f3" exitCode=0 Oct 30 21:38:30.001471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:30.001322 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" event={"ID":"98c9ec3b-435e-4e24-ae23-bf4da48820db","Type":"ContainerDied","Data":"fbf865f476ecb037cfe7aa873ec36362f4ca4ed1526755718cfc57bda37522f3"} Oct 30 21:38:30.248019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:30.247927 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:38:30.248189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:30.248024 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 21:38:31.151356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.151297 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:31.247676 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.247649 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-xks-ca\") pod \"98c9ec3b-435e-4e24-ae23-bf4da48820db\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " Oct 30 21:38:31.247885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.247717 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-openshift-ca\") pod \"98c9ec3b-435e-4e24-ae23-bf4da48820db\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " Oct 30 21:38:31.247885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.247763 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c9ec3b-435e-4e24-ae23-bf4da48820db-kube-api-access-kpxql\" (UniqueName: \"kubernetes.io/projected/98c9ec3b-435e-4e24-ae23-bf4da48820db-kube-api-access-kpxql\") pod \"98c9ec3b-435e-4e24-ae23-bf4da48820db\" (UID: \"98c9ec3b-435e-4e24-ae23-bf4da48820db\") " Oct 30 21:38:31.248133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.248093 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-openshift-ca" pod "98c9ec3b-435e-4e24-ae23-bf4da48820db" (UID: "98c9ec3b-435e-4e24-ae23-bf4da48820db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:31.250191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.250163 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-xks-ca" pod "98c9ec3b-435e-4e24-ae23-bf4da48820db" (UID: "98c9ec3b-435e-4e24-ae23-bf4da48820db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:31.250191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.250173 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c9ec3b-435e-4e24-ae23-bf4da48820db-kube-api-access-kpxql" pod "98c9ec3b-435e-4e24-ae23-bf4da48820db" (UID: "98c9ec3b-435e-4e24-ae23-bf4da48820db"). InnerVolumeSpecName "kube-api-access-kpxql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:31.252361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.252342 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:31.252652 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.252630 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:31.253317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.253301 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:38:31.349758 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.349724 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:31.349758 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.349758 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c9ec3b-435e-4e24-ae23-bf4da48820db-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:31.350014 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:31.349771 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpxql\" (UniqueName: \"kubernetes.io/projected/98c9ec3b-435e-4e24-ae23-bf4da48820db-kube-api-access-kpxql\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:32.016180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:32.016150 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" Oct 30 21:38:32.016356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:32.016180 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl" event={"ID":"98c9ec3b-435e-4e24-ae23-bf4da48820db","Type":"ContainerDied","Data":"ab04067e04ab3fca9f0f893619e1dcebbf9c9415c97cbbb78e73f7baeb910aaa"} Oct 30 21:38:32.016356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:32.016225 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab04067e04ab3fca9f0f893619e1dcebbf9c9415c97cbbb78e73f7baeb910aaa" Oct 30 21:38:34.045009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:34.044971 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww"] Oct 30 21:38:34.094299 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:34.094265 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80a4d1fe-cj4ww"] Oct 30 21:38:35.985778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:35.985745 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd" path="/var/lib/kubelet/pods/a2aa8c58-98c7-41ba-b7f2-0b0842dbf0bd/volumes" Oct 30 21:38:38.616818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.616784 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f"] Oct 30 21:38:38.617335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.617315 4526 cpu_manager.go:414] "containerMap: removing container" podUID="98c9ec3b-435e-4e24-ae23-bf4da48820db" containerName="revoke-keys" Oct 30 21:38:38.617335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.617326 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c9ec3b-435e-4e24-ae23-bf4da48820db" containerName="revoke-keys" Oct 30 21:38:38.617454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.617442 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c9ec3b-435e-4e24-ae23-bf4da48820db" containerName="revoke-keys" Oct 30 21:38:38.620909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.620893 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.637736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.637710 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.637860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.637758 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.637860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.637829 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.637860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.637856 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.637963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.637939 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.638071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.638055 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8rtv\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-kube-api-access-z8rtv\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.638116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.638103 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.638151 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.638142 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.638195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.638185 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739046 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739174 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739220 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739256 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739282 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739310 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739411 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8rtv\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-kube-api-access-z8rtv\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739449 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739531 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739617 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.739798 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.739770 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.740153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.740124 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.740269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.740199 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.740726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.740694 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.741996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.741973 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.742255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.742239 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.795787 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.795744 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.796003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.795865 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8rtv\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-kube-api-access-z8rtv\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:38.833111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.833077 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f"] Oct 30 21:38:38.934456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:38.934358 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:39.386773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:39.386716 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f"] Oct 30 21:38:39.403047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:39.402981 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:39.403212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:39.403084 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:39.403212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:39.403136 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:40.072515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:40.072233 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w"] Oct 30 21:38:40.085542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:40.082818 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" event={"ID":"470b0a52-6093-4cb9-85ef-ce5b3bd93485","Type":"ContainerStarted","Data":"2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60"} Oct 30 21:38:40.085542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:40.082894 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" event={"ID":"470b0a52-6093-4cb9-85ef-ce5b3bd93485","Type":"ContainerStarted","Data":"77cd2b15524790149d2c4a7bd97d8d90877ded116bc590b30ce97ad7cf61939b"} Oct 30 21:38:40.147515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:40.144067 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4938380d-hnf4w"] Oct 30 21:38:40.241515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:40.223138 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podStartSLOduration=2.223115736 podStartE2EDuration="2.223115736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:40.21743107 +0000 UTC m=+1310.350804270" watchObservedRunningTime="2026-10-30 21:38:40.223115736 +0000 UTC m=+1310.356488938" Oct 30 21:38:40.990861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:40.990825 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:41.020436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:40.998453 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:38:41.020849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:41.020811 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:38:41.935579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:41.935542 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:38:41.936098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:41.935610 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:38:41.957810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:41.957780 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh"] Oct 30 21:38:41.961924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:41.961903 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:41.985880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:41.985853 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8900c9c-b546-47e9-afe9-ad4fec778605" path="/var/lib/kubelet/pods/e8900c9c-b546-47e9-afe9-ad4fec778605/volumes" Oct 30 21:38:42.020286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.019619 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:42.064268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.064236 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:42.065333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.065317 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:38:42.065391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.065317 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:42.069265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.069243 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh"] Oct 30 21:38:42.074690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.074668 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.074769 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.074724 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.074918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.074900 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cr6w\" (UniqueName: \"kubernetes.io/projected/085f232e-e45d-4ddf-abdf-95ad40031bf7-kube-api-access-9cr6w\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.176405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.176360 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.176694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.176553 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cr6w\" (UniqueName: \"kubernetes.io/projected/085f232e-e45d-4ddf-abdf-95ad40031bf7-kube-api-access-9cr6w\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.176778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.176695 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.177135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.177112 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.179128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.179110 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.234355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.234278 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cr6w\" (UniqueName: \"kubernetes.io/projected/085f232e-e45d-4ddf-abdf-95ad40031bf7-kube-api-access-9cr6w\") pod \"maas-api-revoke-sub-35d721a8-58nsh\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.277675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.277639 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:42.330822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.330785 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx"] Oct 30 21:38:42.335225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.335194 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.380089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.379171 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.380089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.379280 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.380089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.379393 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lccv\" (UniqueName: \"kubernetes.io/projected/f2e8e269-2272-48de-999c-fdc531cfbbf7-kube-api-access-6lccv\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.420543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.419602 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx"] Oct 30 21:38:42.480186 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:38:42.480149 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:38:42.480403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.480375 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.480515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.480480 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.480612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.480598 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lccv\" (UniqueName: \"kubernetes.io/projected/f2e8e269-2272-48de-999c-fdc531cfbbf7-kube-api-access-6lccv\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.481387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.481362 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.483760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.483738 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.537940 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.537913 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lccv\" (UniqueName: \"kubernetes.io/projected/f2e8e269-2272-48de-999c-fdc531cfbbf7-kube-api-access-6lccv\") pod \"maas-api-revoke-sub-7024597b-6d2lx\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.650683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.650648 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:42.691032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.691004 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh"] Oct 30 21:38:42.691032 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:42.691023 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085f232e_e45d_4ddf_abdf_95ad40031bf7.slice/crio-fa815d387cafb0a238a9710a92b8b12e68dc0b72a922bd21ede127532997d768 WatchSource:0}: Error finding container fa815d387cafb0a238a9710a92b8b12e68dc0b72a922bd21ede127532997d768: Status 404 returned error can't find the container with id fa815d387cafb0a238a9710a92b8b12e68dc0b72a922bd21ede127532997d768 Oct 30 21:38:42.935220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.935147 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:38:42.935220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:42.935198 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:38:43.059837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:43.059811 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx"] Oct 30 21:38:43.060191 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:43.060137 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e8e269_2272_48de_999c_fdc531cfbbf7.slice/crio-69aabc38f859a365cf48a6fcb0ae37aadfc6a6e48ee4a34d137b286393283a7a WatchSource:0}: Error finding container 69aabc38f859a365cf48a6fcb0ae37aadfc6a6e48ee4a34d137b286393283a7a: Status 404 returned error can't find the container with id 69aabc38f859a365cf48a6fcb0ae37aadfc6a6e48ee4a34d137b286393283a7a Oct 30 21:38:43.133403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:43.133372 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" event={"ID":"085f232e-e45d-4ddf-abdf-95ad40031bf7","Type":"ContainerStarted","Data":"fa815d387cafb0a238a9710a92b8b12e68dc0b72a922bd21ede127532997d768"} Oct 30 21:38:43.134673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:43.134651 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" event={"ID":"f2e8e269-2272-48de-999c-fdc531cfbbf7","Type":"ContainerStarted","Data":"69aabc38f859a365cf48a6fcb0ae37aadfc6a6e48ee4a34d137b286393283a7a"} Oct 30 21:38:43.935083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:43.935053 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:38:43.935238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:43.935102 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:38:44.143786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:44.143754 4526 generic.go:356] "Generic (PLEG): container finished" podID="085f232e-e45d-4ddf-abdf-95ad40031bf7" containerID="f2a05687bc39bd25da7e90708106d687aac13c36a3eb2fb5ff96e98adf29bc5d" exitCode=0 Oct 30 21:38:44.144203 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:44.143836 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" event={"ID":"085f232e-e45d-4ddf-abdf-95ad40031bf7","Type":"ContainerDied","Data":"f2a05687bc39bd25da7e90708106d687aac13c36a3eb2fb5ff96e98adf29bc5d"} Oct 30 21:38:44.145428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:44.145408 4526 generic.go:356] "Generic (PLEG): container finished" podID="f2e8e269-2272-48de-999c-fdc531cfbbf7" containerID="b9499d9ebe29eaba174d6a1ecfdd0064b9e8bd0998db3c4b4777ec6778063c44" exitCode=0 Oct 30 21:38:44.145563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:44.145459 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" event={"ID":"f2e8e269-2272-48de-999c-fdc531cfbbf7","Type":"ContainerDied","Data":"b9499d9ebe29eaba174d6a1ecfdd0064b9e8bd0998db3c4b4777ec6778063c44"} Oct 30 21:38:44.935611 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:44.935574 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:38:44.935776 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:44.935635 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:38:45.049293 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.049261 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq"] Oct 30 21:38:45.122524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.122465 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49bee2d1-stpjq"] Oct 30 21:38:45.355671 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.355644 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:45.417514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.415699 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:45.434118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.434090 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-xks-ca\") pod \"085f232e-e45d-4ddf-abdf-95ad40031bf7\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " Oct 30 21:38:45.434255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.434142 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-openshift-ca\") pod \"085f232e-e45d-4ddf-abdf-95ad40031bf7\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " Oct 30 21:38:45.434437 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.434417 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085f232e-e45d-4ddf-abdf-95ad40031bf7-kube-api-access-9cr6w\" (UniqueName: \"kubernetes.io/projected/085f232e-e45d-4ddf-abdf-95ad40031bf7-kube-api-access-9cr6w\") pod \"085f232e-e45d-4ddf-abdf-95ad40031bf7\" (UID: \"085f232e-e45d-4ddf-abdf-95ad40031bf7\") " Oct 30 21:38:45.434586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.434543 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-openshift-ca" pod "085f232e-e45d-4ddf-abdf-95ad40031bf7" (UID: "085f232e-e45d-4ddf-abdf-95ad40031bf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:45.435093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.435061 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:45.436434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.436412 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-xks-ca" pod "085f232e-e45d-4ddf-abdf-95ad40031bf7" (UID: "085f232e-e45d-4ddf-abdf-95ad40031bf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:45.436714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.436699 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085f232e-e45d-4ddf-abdf-95ad40031bf7-kube-api-access-9cr6w" pod "085f232e-e45d-4ddf-abdf-95ad40031bf7" (UID: "085f232e-e45d-4ddf-abdf-95ad40031bf7"). InnerVolumeSpecName "kube-api-access-9cr6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:45.535780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.535743 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e8e269-2272-48de-999c-fdc531cfbbf7-kube-api-access-6lccv\" (UniqueName: \"kubernetes.io/projected/f2e8e269-2272-48de-999c-fdc531cfbbf7-kube-api-access-6lccv\") pod \"f2e8e269-2272-48de-999c-fdc531cfbbf7\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " Oct 30 21:38:45.536246 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.536220 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-xks-ca\") pod \"f2e8e269-2272-48de-999c-fdc531cfbbf7\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " Oct 30 21:38:45.536590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.536570 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-openshift-ca\") pod \"f2e8e269-2272-48de-999c-fdc531cfbbf7\" (UID: \"f2e8e269-2272-48de-999c-fdc531cfbbf7\") " Oct 30 21:38:45.537192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.537164 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-openshift-ca" pod "f2e8e269-2272-48de-999c-fdc531cfbbf7" (UID: "f2e8e269-2272-48de-999c-fdc531cfbbf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:45.537546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.537523 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cr6w\" (UniqueName: \"kubernetes.io/projected/085f232e-e45d-4ddf-abdf-95ad40031bf7-kube-api-access-9cr6w\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:45.537681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.537665 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/085f232e-e45d-4ddf-abdf-95ad40031bf7-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:45.537773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.537761 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:45.538243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.538206 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e8e269-2272-48de-999c-fdc531cfbbf7-kube-api-access-6lccv" pod "f2e8e269-2272-48de-999c-fdc531cfbbf7" (UID: "f2e8e269-2272-48de-999c-fdc531cfbbf7"). InnerVolumeSpecName "kube-api-access-6lccv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:45.538336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.538297 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-xks-ca" pod "f2e8e269-2272-48de-999c-fdc531cfbbf7" (UID: "f2e8e269-2272-48de-999c-fdc531cfbbf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:45.638642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.638610 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lccv\" (UniqueName: \"kubernetes.io/projected/f2e8e269-2272-48de-999c-fdc531cfbbf7-kube-api-access-6lccv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:45.638642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.638640 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e8e269-2272-48de-999c-fdc531cfbbf7-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:45.935639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.935545 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 21:38:45.935786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.935620 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 21:38:45.990243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:45.990164 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a362fbd9-09f8-4bd3-8f87-36af30a08b33" path="/var/lib/kubelet/pods/a362fbd9-09f8-4bd3-8f87-36af30a08b33/volumes" Oct 30 21:38:46.161719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.161687 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" Oct 30 21:38:46.161915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.161719 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh" event={"ID":"085f232e-e45d-4ddf-abdf-95ad40031bf7","Type":"ContainerDied","Data":"fa815d387cafb0a238a9710a92b8b12e68dc0b72a922bd21ede127532997d768"} Oct 30 21:38:46.161915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.161771 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa815d387cafb0a238a9710a92b8b12e68dc0b72a922bd21ede127532997d768" Oct 30 21:38:46.163349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.163330 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" Oct 30 21:38:46.163473 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.163363 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx" event={"ID":"f2e8e269-2272-48de-999c-fdc531cfbbf7","Type":"ContainerDied","Data":"69aabc38f859a365cf48a6fcb0ae37aadfc6a6e48ee4a34d137b286393283a7a"} Oct 30 21:38:46.163473 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.163429 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69aabc38f859a365cf48a6fcb0ae37aadfc6a6e48ee4a34d137b286393283a7a" Oct 30 21:38:46.543527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.543470 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6"] Oct 30 21:38:46.544038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.544021 4526 cpu_manager.go:414] "containerMap: removing container" podUID="085f232e-e45d-4ddf-abdf-95ad40031bf7" containerName="revoke-keys" Oct 30 21:38:46.544038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.544037 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085f232e-e45d-4ddf-abdf-95ad40031bf7" containerName="revoke-keys" Oct 30 21:38:46.544171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.544049 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f2e8e269-2272-48de-999c-fdc531cfbbf7" containerName="revoke-keys" Oct 30 21:38:46.544171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.544055 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e8e269-2272-48de-999c-fdc531cfbbf7" containerName="revoke-keys" Oct 30 21:38:46.544171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.544158 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="085f232e-e45d-4ddf-abdf-95ad40031bf7" containerName="revoke-keys" Oct 30 21:38:46.544171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.544166 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e8e269-2272-48de-999c-fdc531cfbbf7" containerName="revoke-keys" Oct 30 21:38:46.548694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.548673 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.601195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.601114 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:46.601195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.601130 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:38:46.601368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.601216 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:38:46.601368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.601272 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:38:46.615454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.615424 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6"] Oct 30 21:38:46.651296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.651254 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.651539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.651470 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.651638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.651611 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vb9\" (UniqueName: \"kubernetes.io/projected/5d26d0e4-7b64-4f32-8066-57de59d25890-kube-api-access-g4vb9\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.752748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.752715 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.752934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.752892 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.752977 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.752959 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vb9\" (UniqueName: \"kubernetes.io/projected/5d26d0e4-7b64-4f32-8066-57de59d25890-kube-api-access-g4vb9\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.753616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.753525 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.755283 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.755259 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.809877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.809836 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vb9\" (UniqueName: \"kubernetes.io/projected/5d26d0e4-7b64-4f32-8066-57de59d25890-kube-api-access-g4vb9\") pod \"maas-api-revoke-sub-a3b100c9-jzzh6\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:46.862966 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:46.862886 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:47.049467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.049430 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk"] Oct 30 21:38:47.052746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.052722 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d466b5-jr4lk"] Oct 30 21:38:47.246349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.246266 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:47.248135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.248100 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:47.248743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.248722 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:38:47.259737 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.259712 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6"] Oct 30 21:38:47.261316 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:47.261290 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d26d0e4_7b64_4f32_8066_57de59d25890.slice/crio-ac9e36fd8cfe6fde231d6e32e6cfddc6100168a397e982820aa7e5079920a2ec WatchSource:0}: Error finding container ac9e36fd8cfe6fde231d6e32e6cfddc6100168a397e982820aa7e5079920a2ec: Status 404 returned error can't find the container with id ac9e36fd8cfe6fde231d6e32e6cfddc6100168a397e982820aa7e5079920a2ec Oct 30 21:38:47.313875 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.313850 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg"] Oct 30 21:38:47.319100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.319073 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.374975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.374950 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1b5b5f-openshift-default-dockercfg-kzsbh\"" Oct 30 21:38:47.381059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.381028 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg"] Oct 30 21:38:47.462114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462071 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.462287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462123 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.462287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462205 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.462287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462256 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.462389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462366 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.462424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462391 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4mt\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-kube-api-access-8l4mt\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.462424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462411 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.462483 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462443 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff158f4-61fe-4363-a424-7af5412ffbbf-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.462552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.462539 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563310 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563429 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563450 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563470 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563536 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563602 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563621 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4mt\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-kube-api-access-8l4mt\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563645 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563688 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff158f4-61fe-4363-a424-7af5412ffbbf-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563834 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.563854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.563848 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.564375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.564156 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.564375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.564187 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.564375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.564372 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff158f4-61fe-4363-a424-7af5412ffbbf-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.566110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.566083 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.566390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.566368 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.622029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.621999 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.622257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.622212 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4mt\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-kube-api-access-8l4mt\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.644643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.644603 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:47.985935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:47.985900 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76dc0e6c-dad3-4b28-b2fa-89b60c5a3208" path="/var/lib/kubelet/pods/76dc0e6c-dad3-4b28-b2fa-89b60c5a3208/volumes" Oct 30 21:38:48.076506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.076464 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg"] Oct 30 21:38:48.077016 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:48.076990 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff158f4_61fe_4363_a424_7af5412ffbbf.slice/crio-0bfbebbd1881846560a21ea5ce477d0240dc60d5990f57d6b990fb0ea29b2f90 WatchSource:0}: Error finding container 0bfbebbd1881846560a21ea5ce477d0240dc60d5990f57d6b990fb0ea29b2f90: Status 404 returned error can't find the container with id 0bfbebbd1881846560a21ea5ce477d0240dc60d5990f57d6b990fb0ea29b2f90 Oct 30 21:38:48.079520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.079471 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:48.079587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.079553 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:48.079623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.079584 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:48.180537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.180497 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" event={"ID":"5d26d0e4-7b64-4f32-8066-57de59d25890","Type":"ContainerStarted","Data":"0d677012f1f8aa6c5247df8f0523a552f2e942eef1a8b43735f18e63fbaeeb95"} Oct 30 21:38:48.180733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.180573 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" event={"ID":"5d26d0e4-7b64-4f32-8066-57de59d25890","Type":"ContainerStarted","Data":"ac9e36fd8cfe6fde231d6e32e6cfddc6100168a397e982820aa7e5079920a2ec"} Oct 30 21:38:48.182741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.182710 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" event={"ID":"8ff158f4-61fe-4363-a424-7af5412ffbbf","Type":"ContainerStarted","Data":"1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857"} Oct 30 21:38:48.182891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.182758 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" event={"ID":"8ff158f4-61fe-4363-a424-7af5412ffbbf","Type":"ContainerStarted","Data":"0bfbebbd1881846560a21ea5ce477d0240dc60d5990f57d6b990fb0ea29b2f90"} Oct 30 21:38:48.294332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.294267 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" podStartSLOduration=2.294253121 podStartE2EDuration="2.294253121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:48.292039838 +0000 UTC m=+1318.425413037" watchObservedRunningTime="2026-10-30 21:38:48.294253121 +0000 UTC m=+1318.427626319" Oct 30 21:38:48.410336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.410283 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podStartSLOduration=1.410267759 podStartE2EDuration="1.410267759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:48.406270665 +0000 UTC m=+1318.539643865" watchObservedRunningTime="2026-10-30 21:38:48.410267759 +0000 UTC m=+1318.543640956" Oct 30 21:38:48.645374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:48.645192 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:49.191618 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.191581 4526 generic.go:356] "Generic (PLEG): container finished" podID="5d26d0e4-7b64-4f32-8066-57de59d25890" containerID="0d677012f1f8aa6c5247df8f0523a552f2e942eef1a8b43735f18e63fbaeeb95" exitCode=0 Oct 30 21:38:49.191794 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.191666 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" event={"ID":"5d26d0e4-7b64-4f32-8066-57de59d25890","Type":"ContainerDied","Data":"0d677012f1f8aa6c5247df8f0523a552f2e942eef1a8b43735f18e63fbaeeb95"} Oct 30 21:38:49.549818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.549790 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp"] Oct 30 21:38:49.553938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.553924 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.620904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.620804 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp"] Oct 30 21:38:49.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.646750 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:38:49.647268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.646798 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:38:49.690565 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.690534 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hxb\" (UniqueName: \"kubernetes.io/projected/bf7d5b5c-7585-4542-b94d-c7770293efb9-kube-api-access-b6hxb\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.690725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.690591 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.690725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.690653 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.792295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.792255 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hxb\" (UniqueName: \"kubernetes.io/projected/bf7d5b5c-7585-4542-b94d-c7770293efb9-kube-api-access-b6hxb\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.792532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.792333 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.792532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.792358 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.793029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.792994 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.819432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.819353 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.848556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.848529 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hxb\" (UniqueName: \"kubernetes.io/projected/bf7d5b5c-7585-4542-b94d-c7770293efb9-kube-api-access-b6hxb\") pod \"maas-api-revoke-sub-ff811e63-bwvbp\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:49.867164 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:49.867125 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:50.158169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.158102 4526 scope.go:122] "RemoveContainer" containerID="5fd284b67bd274c2a3f409f528ca3a59ee892362321347712531986c3bda2032" Oct 30 21:38:50.170700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.170653 4526 scope.go:122] "RemoveContainer" containerID="112d9fc325fac8bb18c16b7fb6f2000f6369678337c086bbda8354a7d452b9a8" Oct 30 21:38:50.181022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.180992 4526 scope.go:122] "RemoveContainer" containerID="82cabc52db0099f3514d2e0ab5095a2f9f4d3bdb8298c5772c639a97bdfca86c" Oct 30 21:38:50.270280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.270230 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp"] Oct 30 21:38:50.273710 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:50.273689 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7d5b5c_7585_4542_b94d_c7770293efb9.slice/crio-3269f3f7b8095077ab2710e8ad9de8e678ce4cbf48083685931976b1d26fa343 WatchSource:0}: Error finding container 3269f3f7b8095077ab2710e8ad9de8e678ce4cbf48083685931976b1d26fa343: Status 404 returned error can't find the container with id 3269f3f7b8095077ab2710e8ad9de8e678ce4cbf48083685931976b1d26fa343 Oct 30 21:38:50.328513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.325580 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp"] Oct 30 21:38:50.335918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.332134 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.428714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.428632 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lbd\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-kube-api-access-95lbd\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.430040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.430006 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.432746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.430089 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.432746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.430171 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.432746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.430211 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.432746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.430241 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.432746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.430334 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.432746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.430380 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30832c28-5ab0-45af-afd4-54360c2e0c2e-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.432746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.430469 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.435416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.435395 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:50.441514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.441469 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp"] Oct 30 21:38:50.531755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.531702 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-openshift-ca\") pod \"5d26d0e4-7b64-4f32-8066-57de59d25890\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " Oct 30 21:38:50.531755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.531765 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-xks-ca\") pod \"5d26d0e4-7b64-4f32-8066-57de59d25890\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " Oct 30 21:38:50.532043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.531825 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d26d0e4-7b64-4f32-8066-57de59d25890-kube-api-access-g4vb9\" (UniqueName: \"kubernetes.io/projected/5d26d0e4-7b64-4f32-8066-57de59d25890-kube-api-access-g4vb9\") pod \"5d26d0e4-7b64-4f32-8066-57de59d25890\" (UID: \"5d26d0e4-7b64-4f32-8066-57de59d25890\") " Oct 30 21:38:50.532108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532061 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-openshift-ca" pod "5d26d0e4-7b64-4f32-8066-57de59d25890" (UID: "5d26d0e4-7b64-4f32-8066-57de59d25890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:50.532108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532101 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532135 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30832c28-5ab0-45af-afd4-54360c2e0c2e-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532198 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532244 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lbd\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-kube-api-access-95lbd\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532278 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532356 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532666 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532714 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532716 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532748 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.532970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.532889 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:50.533095 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.533072 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.533186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.533101 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30832c28-5ab0-45af-afd4-54360c2e0c2e-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.533228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.533183 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.533352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.533331 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.534320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.534299 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-xks-ca" pod "5d26d0e4-7b64-4f32-8066-57de59d25890" (UID: "5d26d0e4-7b64-4f32-8066-57de59d25890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:50.534821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.534798 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d26d0e4-7b64-4f32-8066-57de59d25890-kube-api-access-g4vb9" pod "5d26d0e4-7b64-4f32-8066-57de59d25890" (UID: "5d26d0e4-7b64-4f32-8066-57de59d25890"). InnerVolumeSpecName "kube-api-access-g4vb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:50.535166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.535150 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.535416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.535402 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.590129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.590094 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lbd\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-kube-api-access-95lbd\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.590312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.590097 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:50.634141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.634109 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d26d0e4-7b64-4f32-8066-57de59d25890-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:50.634141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.634137 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vb9\" (UniqueName: \"kubernetes.io/projected/5d26d0e4-7b64-4f32-8066-57de59d25890-kube-api-access-g4vb9\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:50.645658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.645626 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:38:50.645814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.645686 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:38:50.689118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:50.689027 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:51.157450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.157422 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp"] Oct 30 21:38:51.158342 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:51.158317 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30832c28_5ab0_45af_afd4_54360c2e0c2e.slice/crio-81641d18365038cb364bdb6be99e228ef3f7ce9207b48b68b8408b0f0ae379e0 WatchSource:0}: Error finding container 81641d18365038cb364bdb6be99e228ef3f7ce9207b48b68b8408b0f0ae379e0: Status 404 returned error can't find the container with id 81641d18365038cb364bdb6be99e228ef3f7ce9207b48b68b8408b0f0ae379e0 Oct 30 21:38:51.160967 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.160935 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:51.161039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.161001 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:51.161039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.161031 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:51.237925 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.237882 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" event={"ID":"bf7d5b5c-7585-4542-b94d-c7770293efb9","Type":"ContainerStarted","Data":"eb9483c2c12ab604522ab27b34b2e24dd22222d1a3cf6f5fca2d642f77c9bdd4"} Oct 30 21:38:51.238100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.237947 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" event={"ID":"bf7d5b5c-7585-4542-b94d-c7770293efb9","Type":"ContainerStarted","Data":"3269f3f7b8095077ab2710e8ad9de8e678ce4cbf48083685931976b1d26fa343"} Oct 30 21:38:51.239254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.239227 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" event={"ID":"5d26d0e4-7b64-4f32-8066-57de59d25890","Type":"ContainerDied","Data":"ac9e36fd8cfe6fde231d6e32e6cfddc6100168a397e982820aa7e5079920a2ec"} Oct 30 21:38:51.239367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.239272 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac9e36fd8cfe6fde231d6e32e6cfddc6100168a397e982820aa7e5079920a2ec" Oct 30 21:38:51.239367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.239329 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6" Oct 30 21:38:51.244094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.244071 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" event={"ID":"30832c28-5ab0-45af-afd4-54360c2e0c2e","Type":"ContainerStarted","Data":"81641d18365038cb364bdb6be99e228ef3f7ce9207b48b68b8408b0f0ae379e0"} Oct 30 21:38:51.354528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.353836 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" podStartSLOduration=2.3538183520000002 podStartE2EDuration="2.353818352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:51.349124571 +0000 UTC m=+1321.482497773" watchObservedRunningTime="2026-10-30 21:38:51.353818352 +0000 UTC m=+1321.487191554" Oct 30 21:38:51.645805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.645770 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:38:51.645987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:51.645837 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:38:52.258354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.258323 4526 generic.go:356] "Generic (PLEG): container finished" podID="bf7d5b5c-7585-4542-b94d-c7770293efb9" containerID="eb9483c2c12ab604522ab27b34b2e24dd22222d1a3cf6f5fca2d642f77c9bdd4" exitCode=0 Oct 30 21:38:52.258785 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.258412 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" event={"ID":"bf7d5b5c-7585-4542-b94d-c7770293efb9","Type":"ContainerDied","Data":"eb9483c2c12ab604522ab27b34b2e24dd22222d1a3cf6f5fca2d642f77c9bdd4"} Oct 30 21:38:52.260076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.260053 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" event={"ID":"30832c28-5ab0-45af-afd4-54360c2e0c2e","Type":"ContainerStarted","Data":"3c040f292e80f56c2fd164d1c5c2368a5764541eff2d8d04bd7996c166348405"} Oct 30 21:38:52.549818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.549734 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podStartSLOduration=2.5497134150000003 podStartE2EDuration="2.549713415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:52.524878813 +0000 UTC m=+1322.658252016" watchObservedRunningTime="2026-10-30 21:38:52.549713415 +0000 UTC m=+1322.683086613" Oct 30 21:38:52.646108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.646060 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:38:52.646295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.646131 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:38:52.690094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.690066 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:52.691913 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.691888 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:38:52.692022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:52.691949 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 21:38:53.582506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.582461 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:53.645184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.645139 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:38:53.645366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.645224 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:38:53.688352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.688313 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n"] Oct 30 21:38:53.689056 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.689037 4526 cpu_manager.go:414] "containerMap: removing container" podUID="5d26d0e4-7b64-4f32-8066-57de59d25890" containerName="revoke-keys" Oct 30 21:38:53.689135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.689059 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d26d0e4-7b64-4f32-8066-57de59d25890" containerName="revoke-keys" Oct 30 21:38:53.689135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.689069 4526 cpu_manager.go:414] "containerMap: removing container" podUID="bf7d5b5c-7585-4542-b94d-c7770293efb9" containerName="revoke-keys" Oct 30 21:38:53.689135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.689074 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7d5b5c-7585-4542-b94d-c7770293efb9" containerName="revoke-keys" Oct 30 21:38:53.689223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.689190 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d26d0e4-7b64-4f32-8066-57de59d25890" containerName="revoke-keys" Oct 30 21:38:53.689223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.689200 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7d5b5c-7585-4542-b94d-c7770293efb9" containerName="revoke-keys" Oct 30 21:38:53.689691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.689670 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:38:53.689741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.689719 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 21:38:53.692592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.692579 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:53.766518 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.765964 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n"] Oct 30 21:38:53.772209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.772182 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-openshift-ca\") pod \"bf7d5b5c-7585-4542-b94d-c7770293efb9\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " Oct 30 21:38:53.772351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.772234 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-xks-ca\") pod \"bf7d5b5c-7585-4542-b94d-c7770293efb9\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " Oct 30 21:38:53.772351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.772267 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7d5b5c-7585-4542-b94d-c7770293efb9-kube-api-access-b6hxb\" (UniqueName: \"kubernetes.io/projected/bf7d5b5c-7585-4542-b94d-c7770293efb9-kube-api-access-b6hxb\") pod \"bf7d5b5c-7585-4542-b94d-c7770293efb9\" (UID: \"bf7d5b5c-7585-4542-b94d-c7770293efb9\") " Oct 30 21:38:53.773044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.773016 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-openshift-ca" pod "bf7d5b5c-7585-4542-b94d-c7770293efb9" (UID: "bf7d5b5c-7585-4542-b94d-c7770293efb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:53.775354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.775317 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-xks-ca" pod "bf7d5b5c-7585-4542-b94d-c7770293efb9" (UID: "bf7d5b5c-7585-4542-b94d-c7770293efb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:53.775544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.775517 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7d5b5c-7585-4542-b94d-c7770293efb9-kube-api-access-b6hxb" pod "bf7d5b5c-7585-4542-b94d-c7770293efb9" (UID: "bf7d5b5c-7585-4542-b94d-c7770293efb9"). InnerVolumeSpecName "kube-api-access-b6hxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:53.873842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.873753 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:53.873842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.873813 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978kp\" (UniqueName: \"kubernetes.io/projected/f9e31e45-a29d-4571-ba6d-b04fe08c9169-kube-api-access-978kp\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:53.874011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.873850 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:53.874011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.873962 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:53.874011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.873973 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d5b5c-7585-4542-b94d-c7770293efb9-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:53.874011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.873982 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6hxb\" (UniqueName: \"kubernetes.io/projected/bf7d5b5c-7585-4542-b94d-c7770293efb9-kube-api-access-b6hxb\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:53.975848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.975807 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:53.976052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.975879 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978kp\" (UniqueName: \"kubernetes.io/projected/f9e31e45-a29d-4571-ba6d-b04fe08c9169-kube-api-access-978kp\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:53.976052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.975927 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:53.976654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.976632 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:53.980121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:53.980090 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:54.044240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.044180 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-978kp\" (UniqueName: \"kubernetes.io/projected/f9e31e45-a29d-4571-ba6d-b04fe08c9169-kube-api-access-978kp\") pod \"maas-api-revoke-sub-336f566e-vb55n\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:54.071859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.071822 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:54.282404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.279578 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" Oct 30 21:38:54.282404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.280083 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp" event={"ID":"bf7d5b5c-7585-4542-b94d-c7770293efb9","Type":"ContainerDied","Data":"3269f3f7b8095077ab2710e8ad9de8e678ce4cbf48083685931976b1d26fa343"} Oct 30 21:38:54.282404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.280128 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3269f3f7b8095077ab2710e8ad9de8e678ce4cbf48083685931976b1d26fa343" Oct 30 21:38:54.525798 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.525690 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n"] Oct 30 21:38:54.645136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.645100 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:38:54.645642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.645159 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 21:38:54.689912 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.689873 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:38:54.690101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:54.689925 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 21:38:55.287888 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:55.287841 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" event={"ID":"f9e31e45-a29d-4571-ba6d-b04fe08c9169","Type":"ContainerStarted","Data":"592d9707ee1149df8e8abbd2a3acbb90d5205765f1cd1310153fa0c0f40780e5"} Oct 30 21:38:55.288093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:55.287916 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" event={"ID":"f9e31e45-a29d-4571-ba6d-b04fe08c9169","Type":"ContainerStarted","Data":"81adeaa10d9a032218bc272d91f4a69aafffedc2b98703ff0623121cb0b35192"} Oct 30 21:38:55.419514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:55.410380 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" podStartSLOduration=2.4103546270000002 podStartE2EDuration="2.410354627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:55.399595526 +0000 UTC m=+1325.532968728" watchObservedRunningTime="2026-10-30 21:38:55.410354627 +0000 UTC m=+1325.543727827" Oct 30 21:38:55.689500 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:55.689453 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:38:55.689881 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:55.689535 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 21:38:55.940771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:55.939792 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:55.940944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:55.940820 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:55.941724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:55.941696 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:38:56.326319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:56.326279 4526 generic.go:356] "Generic (PLEG): container finished" podID="f9e31e45-a29d-4571-ba6d-b04fe08c9169" containerID="592d9707ee1149df8e8abbd2a3acbb90d5205765f1cd1310153fa0c0f40780e5" exitCode=0 Oct 30 21:38:56.326525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:56.326364 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" event={"ID":"f9e31e45-a29d-4571-ba6d-b04fe08c9169","Type":"ContainerDied","Data":"592d9707ee1149df8e8abbd2a3acbb90d5205765f1cd1310153fa0c0f40780e5"} Oct 30 21:38:56.689555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:56.689468 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:38:56.689923 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:56.689550 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 21:38:57.045844 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.045817 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk"] Oct 30 21:38:57.049268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.049248 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70412924-cgxgk"] Oct 30 21:38:57.305967 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.305881 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq"] Oct 30 21:38:57.310219 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.310202 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.343924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.343885 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.344152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.344124 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.344420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.344328 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7h8\" (UniqueName: \"kubernetes.io/projected/d14a218f-25ae-4f71-9800-917eb5c993db-kube-api-access-8x7h8\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.364818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.364766 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq"] Oct 30 21:38:57.445394 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.445360 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.445601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.445402 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.445601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.445562 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7h8\" (UniqueName: \"kubernetes.io/projected/d14a218f-25ae-4f71-9800-917eb5c993db-kube-api-access-8x7h8\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.448231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.448198 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.459028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.459001 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.504892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.504869 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7h8\" (UniqueName: \"kubernetes.io/projected/d14a218f-25ae-4f71-9800-917eb5c993db-kube-api-access-8x7h8\") pod \"maas-api-revoke-sub-5ed1b355-tlnkq\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.512653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.512629 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6"] Oct 30 21:38:57.517993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.517975 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.545518 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.545498 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:57.546119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.546101 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-openshift-ca\") pod \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " Oct 30 21:38:57.546232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.546132 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e31e45-a29d-4571-ba6d-b04fe08c9169-kube-api-access-978kp\" (UniqueName: \"kubernetes.io/projected/f9e31e45-a29d-4571-ba6d-b04fe08c9169-kube-api-access-978kp\") pod \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " Oct 30 21:38:57.546232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.546172 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-xks-ca\") pod \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\" (UID: \"f9e31e45-a29d-4571-ba6d-b04fe08c9169\") " Oct 30 21:38:57.546423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.546409 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.546566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.546544 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.546566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.546543 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-openshift-ca" pod "f9e31e45-a29d-4571-ba6d-b04fe08c9169" (UID: "f9e31e45-a29d-4571-ba6d-b04fe08c9169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:57.546729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.546639 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvq4z\" (UniqueName: \"kubernetes.io/projected/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-kube-api-access-jvq4z\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.546881 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.546865 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:57.549428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.549406 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-xks-ca" pod "f9e31e45-a29d-4571-ba6d-b04fe08c9169" (UID: "f9e31e45-a29d-4571-ba6d-b04fe08c9169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:57.549537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.549433 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e31e45-a29d-4571-ba6d-b04fe08c9169-kube-api-access-978kp" pod "f9e31e45-a29d-4571-ba6d-b04fe08c9169" (UID: "f9e31e45-a29d-4571-ba6d-b04fe08c9169"). InnerVolumeSpecName "kube-api-access-978kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:57.574960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.574937 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6"] Oct 30 21:38:57.623585 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.623554 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:38:57.648020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.647993 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.648127 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.648047 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvq4z\" (UniqueName: \"kubernetes.io/projected/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-kube-api-access-jvq4z\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.648235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.648221 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.648324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.648309 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-978kp\" (UniqueName: \"kubernetes.io/projected/f9e31e45-a29d-4571-ba6d-b04fe08c9169-kube-api-access-978kp\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:57.648399 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.648331 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e31e45-a29d-4571-ba6d-b04fe08c9169-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:38:57.648882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.648859 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.650653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.650632 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.690099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.690069 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:38:57.690521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.690116 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 21:38:57.705896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.705872 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvq4z\" (UniqueName: \"kubernetes.io/projected/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-kube-api-access-jvq4z\") pod \"maas-api-revoke-sub-09025997-7zmj6\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.864704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.864610 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:38:57.985771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:57.985738 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3996407b-fd70-401a-a329-ad9ab6542ef9" path="/var/lib/kubelet/pods/3996407b-fd70-401a-a329-ad9ab6542ef9/volumes" Oct 30 21:38:58.007958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.007933 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq"] Oct 30 21:38:58.008167 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:58.008149 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14a218f_25ae_4f71_9800_917eb5c993db.slice/crio-168c274af1a9e9b982051db99f333339522dfe1ee6048a8854fe6970887c8f21 WatchSource:0}: Error finding container 168c274af1a9e9b982051db99f333339522dfe1ee6048a8854fe6970887c8f21: Status 404 returned error can't find the container with id 168c274af1a9e9b982051db99f333339522dfe1ee6048a8854fe6970887c8f21 Oct 30 21:38:58.248634 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:38:58.248584 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9daaca_776d_4d69_9cbf_4d96f1d0dd1d.slice/crio-81c00adca75dd8d9ce301da6af2b166072fd54d758ba82a8ddbe01e9827cda41 WatchSource:0}: Error finding container 81c00adca75dd8d9ce301da6af2b166072fd54d758ba82a8ddbe01e9827cda41: Status 404 returned error can't find the container with id 81c00adca75dd8d9ce301da6af2b166072fd54d758ba82a8ddbe01e9827cda41 Oct 30 21:38:58.249002 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.248977 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6"] Oct 30 21:38:58.352271 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.352230 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" event={"ID":"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d","Type":"ContainerStarted","Data":"81c00adca75dd8d9ce301da6af2b166072fd54d758ba82a8ddbe01e9827cda41"} Oct 30 21:38:58.354434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.354407 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" Oct 30 21:38:58.354626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.354508 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n" event={"ID":"f9e31e45-a29d-4571-ba6d-b04fe08c9169","Type":"ContainerDied","Data":"81adeaa10d9a032218bc272d91f4a69aafffedc2b98703ff0623121cb0b35192"} Oct 30 21:38:58.354626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.354565 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81adeaa10d9a032218bc272d91f4a69aafffedc2b98703ff0623121cb0b35192" Oct 30 21:38:58.360158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.360131 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" event={"ID":"d14a218f-25ae-4f71-9800-917eb5c993db","Type":"ContainerStarted","Data":"28754d3c4cd76f8ae22faa856455b523a39170c475a4fb9a4ee8d544c1bb82d8"} Oct 30 21:38:58.360332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.360316 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" event={"ID":"d14a218f-25ae-4f71-9800-917eb5c993db","Type":"ContainerStarted","Data":"168c274af1a9e9b982051db99f333339522dfe1ee6048a8854fe6970887c8f21"} Oct 30 21:38:58.526726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.526660 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" podStartSLOduration=1.5266424440000002 podStartE2EDuration="1.526642444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:58.523329348 +0000 UTC m=+1328.656702547" watchObservedRunningTime="2026-10-30 21:38:58.526642444 +0000 UTC m=+1328.660015647" Oct 30 21:38:58.749777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.749749 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:58.750580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.750552 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:58.751514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:58.751476 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:38:59.371239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:59.371209 4526 generic.go:356] "Generic (PLEG): container finished" podID="6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d" containerID="9fbc19b1cac1e0adcc67ba73917291ba48506f6fd3d40a3fda949fdc8829ef68" exitCode=0 Oct 30 21:38:59.371427 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:59.371289 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" event={"ID":"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d","Type":"ContainerDied","Data":"9fbc19b1cac1e0adcc67ba73917291ba48506f6fd3d40a3fda949fdc8829ef68"} Oct 30 21:38:59.373423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:59.373398 4526 generic.go:356] "Generic (PLEG): container finished" podID="d14a218f-25ae-4f71-9800-917eb5c993db" containerID="28754d3c4cd76f8ae22faa856455b523a39170c475a4fb9a4ee8d544c1bb82d8" exitCode=0 Oct 30 21:38:59.373566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:38:59.373481 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" event={"ID":"d14a218f-25ae-4f71-9800-917eb5c993db","Type":"ContainerDied","Data":"28754d3c4cd76f8ae22faa856455b523a39170c475a4fb9a4ee8d544c1bb82d8"} Oct 30 21:39:00.571467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.571441 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:39:00.575665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.575641 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:39:00.583499 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.583463 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-xks-ca\") pod \"d14a218f-25ae-4f71-9800-917eb5c993db\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " Oct 30 21:39:00.583619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.583586 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14a218f-25ae-4f71-9800-917eb5c993db-kube-api-access-8x7h8\" (UniqueName: \"kubernetes.io/projected/d14a218f-25ae-4f71-9800-917eb5c993db-kube-api-access-8x7h8\") pod \"d14a218f-25ae-4f71-9800-917eb5c993db\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " Oct 30 21:39:00.583802 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.583685 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-openshift-ca\") pod \"d14a218f-25ae-4f71-9800-917eb5c993db\" (UID: \"d14a218f-25ae-4f71-9800-917eb5c993db\") " Oct 30 21:39:00.584047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.584022 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-openshift-ca" pod "d14a218f-25ae-4f71-9800-917eb5c993db" (UID: "d14a218f-25ae-4f71-9800-917eb5c993db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:00.584274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.584258 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.585779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.585759 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14a218f-25ae-4f71-9800-917eb5c993db-kube-api-access-8x7h8" pod "d14a218f-25ae-4f71-9800-917eb5c993db" (UID: "d14a218f-25ae-4f71-9800-917eb5c993db"). InnerVolumeSpecName "kube-api-access-8x7h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:00.585911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.585893 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-xks-ca" pod "d14a218f-25ae-4f71-9800-917eb5c993db" (UID: "d14a218f-25ae-4f71-9800-917eb5c993db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:00.684779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.684694 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-openshift-ca\") pod \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " Oct 30 21:39:00.684779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.684739 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-xks-ca\") pod \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " Oct 30 21:39:00.685017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.684799 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-kube-api-access-jvq4z\" (UniqueName: \"kubernetes.io/projected/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-kube-api-access-jvq4z\") pod \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\" (UID: \"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d\") " Oct 30 21:39:00.685111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.685084 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-openshift-ca" pod "6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d" (UID: "6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:00.685467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.685440 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7h8\" (UniqueName: \"kubernetes.io/projected/d14a218f-25ae-4f71-9800-917eb5c993db-kube-api-access-8x7h8\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.685640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.685471 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.685640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.685505 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14a218f-25ae-4f71-9800-917eb5c993db-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.688285 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.687980 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-xks-ca" pod "6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d" (UID: "6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:00.688955 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.688669 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-kube-api-access-jvq4z" pod "6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d" (UID: "6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d"). InnerVolumeSpecName "kube-api-access-jvq4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:00.786516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.786456 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:00.786694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:00.786524 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvq4z\" (UniqueName: \"kubernetes.io/projected/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d-kube-api-access-jvq4z\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:01.012974 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.012886 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg"] Oct 30 21:39:01.013479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013461 4526 cpu_manager.go:414] "containerMap: removing container" podUID="d14a218f-25ae-4f71-9800-917eb5c993db" containerName="revoke-keys" Oct 30 21:39:01.013546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013483 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14a218f-25ae-4f71-9800-917eb5c993db" containerName="revoke-keys" Oct 30 21:39:01.013546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013518 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f9e31e45-a29d-4571-ba6d-b04fe08c9169" containerName="revoke-keys" Oct 30 21:39:01.013546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013523 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e31e45-a29d-4571-ba6d-b04fe08c9169" containerName="revoke-keys" Oct 30 21:39:01.013643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013561 4526 cpu_manager.go:414] "containerMap: removing container" podUID="6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d" containerName="revoke-keys" Oct 30 21:39:01.013643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013568 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d" containerName="revoke-keys" Oct 30 21:39:01.013703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013674 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e31e45-a29d-4571-ba6d-b04fe08c9169" containerName="revoke-keys" Oct 30 21:39:01.013703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013686 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d" containerName="revoke-keys" Oct 30 21:39:01.013703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.013695 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14a218f-25ae-4f71-9800-917eb5c993db" containerName="revoke-keys" Oct 30 21:39:01.017410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.017395 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.046653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.046622 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8"] Oct 30 21:39:01.050024 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.050003 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f26399c-bpnk8"] Oct 30 21:39:01.074689 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.074655 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg"] Oct 30 21:39:01.090022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.089984 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxxf\" (UniqueName: \"kubernetes.io/projected/97648b85-6605-4a9a-9be1-d4117b648245-kube-api-access-mrxxf\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.090222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.090090 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.090290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.090224 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.191863 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.191829 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxxf\" (UniqueName: \"kubernetes.io/projected/97648b85-6605-4a9a-9be1-d4117b648245-kube-api-access-mrxxf\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.192035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.191874 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.192035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.191944 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.192730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.192705 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.194469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.194453 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.248975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.248945 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxxf\" (UniqueName: \"kubernetes.io/projected/97648b85-6605-4a9a-9be1-d4117b648245-kube-api-access-mrxxf\") pod \"maas-api-revoke-sub-ec10f526-sdlhg\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.331766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.331736 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:01.399512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.393035 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" Oct 30 21:39:01.399512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.396340 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq" event={"ID":"d14a218f-25ae-4f71-9800-917eb5c993db","Type":"ContainerDied","Data":"168c274af1a9e9b982051db99f333339522dfe1ee6048a8854fe6970887c8f21"} Oct 30 21:39:01.399512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.396402 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168c274af1a9e9b982051db99f333339522dfe1ee6048a8854fe6970887c8f21" Oct 30 21:39:01.404010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.402618 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" event={"ID":"6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d","Type":"ContainerDied","Data":"81c00adca75dd8d9ce301da6af2b166072fd54d758ba82a8ddbe01e9827cda41"} Oct 30 21:39:01.404010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.402661 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81c00adca75dd8d9ce301da6af2b166072fd54d758ba82a8ddbe01e9827cda41" Oct 30 21:39:01.404010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.402729 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6" Oct 30 21:39:01.710200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.710163 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg"] Oct 30 21:39:01.710535 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:01.710279 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97648b85_6605_4a9a_9be1_d4117b648245.slice/crio-8fbc5b4a682c1f75f27c20b5fa3e185d927974c8f6010c47d5bf3171b64d1c68 WatchSource:0}: Error finding container 8fbc5b4a682c1f75f27c20b5fa3e185d927974c8f6010c47d5bf3171b64d1c68: Status 404 returned error can't find the container with id 8fbc5b4a682c1f75f27c20b5fa3e185d927974c8f6010c47d5bf3171b64d1c68 Oct 30 21:39:01.986361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:01.986285 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec58bb5-7c6f-4190-936c-57f6b01720dd" path="/var/lib/kubelet/pods/9ec58bb5-7c6f-4190-936c-57f6b01720dd/volumes" Oct 30 21:39:02.412349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:02.412314 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" event={"ID":"97648b85-6605-4a9a-9be1-d4117b648245","Type":"ContainerStarted","Data":"e839a2f4f9dff60b2994e89f909af227ea569308d75ab44e4d69b7995180d6fa"} Oct 30 21:39:02.412556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:02.412366 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" event={"ID":"97648b85-6605-4a9a-9be1-d4117b648245","Type":"ContainerStarted","Data":"8fbc5b4a682c1f75f27c20b5fa3e185d927974c8f6010c47d5bf3171b64d1c68"} Oct 30 21:39:02.525948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:02.525892 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" podStartSLOduration=2.525876294 podStartE2EDuration="2.525876294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:02.524865967 +0000 UTC m=+1332.658239165" watchObservedRunningTime="2026-10-30 21:39:02.525876294 +0000 UTC m=+1332.659249535" Oct 30 21:39:03.054535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:03.054503 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc"] Oct 30 21:39:03.103390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:03.103351 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c216b558-8nvwc"] Oct 30 21:39:03.422226 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:03.422134 4526 generic.go:356] "Generic (PLEG): container finished" podID="97648b85-6605-4a9a-9be1-d4117b648245" containerID="e839a2f4f9dff60b2994e89f909af227ea569308d75ab44e4d69b7995180d6fa" exitCode=0 Oct 30 21:39:03.422382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:03.422220 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" event={"ID":"97648b85-6605-4a9a-9be1-d4117b648245","Type":"ContainerDied","Data":"e839a2f4f9dff60b2994e89f909af227ea569308d75ab44e4d69b7995180d6fa"} Oct 30 21:39:03.986227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:03.986194 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45438f16-6480-4add-aa65-5f5e8a6865bb" path="/var/lib/kubelet/pods/45438f16-6480-4add-aa65-5f5e8a6865bb/volumes" Oct 30 21:39:04.407789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.407750 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh"] Oct 30 21:39:04.412231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.412204 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.467342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.467314 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh"] Oct 30 21:39:04.536785 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.536747 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzb46\" (UniqueName: \"kubernetes.io/projected/77a705f6-736d-42d6-a03c-33000cf92048-kube-api-access-gzb46\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.537017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.536924 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.537221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.537200 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.603589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.603535 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:04.638329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.638296 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.638538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.638439 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzb46\" (UniqueName: \"kubernetes.io/projected/77a705f6-736d-42d6-a03c-33000cf92048-kube-api-access-gzb46\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.638605 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.638553 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.639068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.639027 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.641211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.641180 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.695395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.695322 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzb46\" (UniqueName: \"kubernetes.io/projected/77a705f6-736d-42d6-a03c-33000cf92048-kube-api-access-gzb46\") pod \"maas-api-revoke-sub-d1bb153b-qr2lh\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.726001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.725975 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:04.739446 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.739420 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-openshift-ca\") pod \"97648b85-6605-4a9a-9be1-d4117b648245\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " Oct 30 21:39:04.739623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.739517 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97648b85-6605-4a9a-9be1-d4117b648245-kube-api-access-mrxxf\" (UniqueName: \"kubernetes.io/projected/97648b85-6605-4a9a-9be1-d4117b648245-kube-api-access-mrxxf\") pod \"97648b85-6605-4a9a-9be1-d4117b648245\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " Oct 30 21:39:04.739691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.739630 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-xks-ca\") pod \"97648b85-6605-4a9a-9be1-d4117b648245\" (UID: \"97648b85-6605-4a9a-9be1-d4117b648245\") " Oct 30 21:39:04.740360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.739930 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-openshift-ca" pod "97648b85-6605-4a9a-9be1-d4117b648245" (UID: "97648b85-6605-4a9a-9be1-d4117b648245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:04.740360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.740327 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:04.742653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.742618 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97648b85-6605-4a9a-9be1-d4117b648245-kube-api-access-mrxxf" pod "97648b85-6605-4a9a-9be1-d4117b648245" (UID: "97648b85-6605-4a9a-9be1-d4117b648245"). InnerVolumeSpecName "kube-api-access-mrxxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:04.742755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.742675 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-xks-ca" pod "97648b85-6605-4a9a-9be1-d4117b648245" (UID: "97648b85-6605-4a9a-9be1-d4117b648245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:04.841441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.841408 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrxxf\" (UniqueName: \"kubernetes.io/projected/97648b85-6605-4a9a-9be1-d4117b648245-kube-api-access-mrxxf\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:04.841441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:04.841434 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97648b85-6605-4a9a-9be1-d4117b648245-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:05.113337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:05.113294 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh"] Oct 30 21:39:05.113655 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:05.113632 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a705f6_736d_42d6_a03c_33000cf92048.slice/crio-1e5c297ba3fcee7f6147dccf74d38ab521bc34e8c56e226d13d9fdeeaf4e11f4 WatchSource:0}: Error finding container 1e5c297ba3fcee7f6147dccf74d38ab521bc34e8c56e226d13d9fdeeaf4e11f4: Status 404 returned error can't find the container with id 1e5c297ba3fcee7f6147dccf74d38ab521bc34e8c56e226d13d9fdeeaf4e11f4 Oct 30 21:39:05.438676 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:05.438639 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" event={"ID":"77a705f6-736d-42d6-a03c-33000cf92048","Type":"ContainerStarted","Data":"f53704ebe574a0f89c1a83dfd43515374a8186a978ea01011822eee9f829df1a"} Oct 30 21:39:05.439207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:05.438738 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" event={"ID":"77a705f6-736d-42d6-a03c-33000cf92048","Type":"ContainerStarted","Data":"1e5c297ba3fcee7f6147dccf74d38ab521bc34e8c56e226d13d9fdeeaf4e11f4"} Oct 30 21:39:05.440252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:05.440233 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" Oct 30 21:39:05.440363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:05.440229 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg" event={"ID":"97648b85-6605-4a9a-9be1-d4117b648245","Type":"ContainerDied","Data":"8fbc5b4a682c1f75f27c20b5fa3e185d927974c8f6010c47d5bf3171b64d1c68"} Oct 30 21:39:05.440363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:05.440355 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fbc5b4a682c1f75f27c20b5fa3e185d927974c8f6010c47d5bf3171b64d1c68" Oct 30 21:39:05.553098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:05.553041 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" podStartSLOduration=1.5530274039999998 podStartE2EDuration="1.553027404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:05.551590872 +0000 UTC m=+1335.684964070" watchObservedRunningTime="2026-10-30 21:39:05.553027404 +0000 UTC m=+1335.686400598" Oct 30 21:39:06.449959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:06.449926 4526 generic.go:356] "Generic (PLEG): container finished" podID="77a705f6-736d-42d6-a03c-33000cf92048" containerID="f53704ebe574a0f89c1a83dfd43515374a8186a978ea01011822eee9f829df1a" exitCode=0 Oct 30 21:39:06.450367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:06.450008 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" event={"ID":"77a705f6-736d-42d6-a03c-33000cf92048","Type":"ContainerDied","Data":"f53704ebe574a0f89c1a83dfd43515374a8186a978ea01011822eee9f829df1a"} Oct 30 21:39:07.047029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.046995 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w"] Oct 30 21:39:07.050760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.050734 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-122eae23-mqg2w"] Oct 30 21:39:07.621476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.621447 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:07.678763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.678726 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a705f6-736d-42d6-a03c-33000cf92048-kube-api-access-gzb46\" (UniqueName: \"kubernetes.io/projected/77a705f6-736d-42d6-a03c-33000cf92048-kube-api-access-gzb46\") pod \"77a705f6-736d-42d6-a03c-33000cf92048\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " Oct 30 21:39:07.678952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.678938 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-openshift-ca\") pod \"77a705f6-736d-42d6-a03c-33000cf92048\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " Oct 30 21:39:07.678992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.678974 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-xks-ca\") pod \"77a705f6-736d-42d6-a03c-33000cf92048\" (UID: \"77a705f6-736d-42d6-a03c-33000cf92048\") " Oct 30 21:39:07.679319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.679289 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-openshift-ca" pod "77a705f6-736d-42d6-a03c-33000cf92048" (UID: "77a705f6-736d-42d6-a03c-33000cf92048"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:07.679566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.679477 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:07.681185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.681156 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a705f6-736d-42d6-a03c-33000cf92048-kube-api-access-gzb46" pod "77a705f6-736d-42d6-a03c-33000cf92048" (UID: "77a705f6-736d-42d6-a03c-33000cf92048"). InnerVolumeSpecName "kube-api-access-gzb46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:07.681308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.681284 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-xks-ca" pod "77a705f6-736d-42d6-a03c-33000cf92048" (UID: "77a705f6-736d-42d6-a03c-33000cf92048"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:07.781106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.781002 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a705f6-736d-42d6-a03c-33000cf92048-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:07.781106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.781037 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzb46\" (UniqueName: \"kubernetes.io/projected/77a705f6-736d-42d6-a03c-33000cf92048-kube-api-access-gzb46\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:07.985648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:07.985610 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc159488-381b-4c46-8add-de64f8b8296a" path="/var/lib/kubelet/pods/bc159488-381b-4c46-8add-de64f8b8296a/volumes" Oct 30 21:39:08.048096 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.048066 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz"] Oct 30 21:39:08.051744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.051722 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a945c0fe-fccvz"] Oct 30 21:39:08.288141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.288110 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb"] Oct 30 21:39:08.288781 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.288761 4526 cpu_manager.go:414] "containerMap: removing container" podUID="97648b85-6605-4a9a-9be1-d4117b648245" containerName="revoke-keys" Oct 30 21:39:08.288781 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.288781 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97648b85-6605-4a9a-9be1-d4117b648245" containerName="revoke-keys" Oct 30 21:39:08.288969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.288799 4526 cpu_manager.go:414] "containerMap: removing container" podUID="77a705f6-736d-42d6-a03c-33000cf92048" containerName="revoke-keys" Oct 30 21:39:08.288969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.288807 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a705f6-736d-42d6-a03c-33000cf92048" containerName="revoke-keys" Oct 30 21:39:08.289092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.288988 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="97648b85-6605-4a9a-9be1-d4117b648245" containerName="revoke-keys" Oct 30 21:39:08.289092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.289002 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a705f6-736d-42d6-a03c-33000cf92048" containerName="revoke-keys" Oct 30 21:39:08.292807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.292785 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.390729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390023 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.390729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390103 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.390729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390133 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.390729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390218 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.390729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390357 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.390729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390383 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhg5\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-kube-api-access-rhhg5\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.390729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390429 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.391209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390818 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.391209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.390845 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.391924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.391878 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb"] Oct 30 21:39:08.467401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.467369 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" Oct 30 21:39:08.467401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.467386 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh" event={"ID":"77a705f6-736d-42d6-a03c-33000cf92048","Type":"ContainerDied","Data":"1e5c297ba3fcee7f6147dccf74d38ab521bc34e8c56e226d13d9fdeeaf4e11f4"} Oct 30 21:39:08.467659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.467427 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5c297ba3fcee7f6147dccf74d38ab521bc34e8c56e226d13d9fdeeaf4e11f4" Oct 30 21:39:08.492416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492372 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.492416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492423 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.492733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492517 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.492733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492662 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.492733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492690 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhg5\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-kube-api-access-rhhg5\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.492906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492738 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.492906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492794 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.492906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492820 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.493055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492914 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.493055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492915 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.493055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.492957 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.493204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.493061 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.493289 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.493259 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.493603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.493583 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.495625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.495604 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.495762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.495745 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.549695 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.549660 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.549869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.549800 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhg5\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-kube-api-access-rhhg5\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:08.606819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:08.606780 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:09.046785 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.046755 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb"] Oct 30 21:39:09.047172 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:09.047079 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d3c749_91d6_4ab8_b1d3_5a6f27c1b076.slice/crio-0f4462bc15f96623137f672889b3a338379a67e6bf66c118d20dd456009c77d8 WatchSource:0}: Error finding container 0f4462bc15f96623137f672889b3a338379a67e6bf66c118d20dd456009c77d8: Status 404 returned error can't find the container with id 0f4462bc15f96623137f672889b3a338379a67e6bf66c118d20dd456009c77d8 Oct 30 21:39:09.049479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.049451 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:09.049572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.049557 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:09.049607 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.049592 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:09.477346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.477230 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" event={"ID":"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076","Type":"ContainerStarted","Data":"b69daf3dd4f1f486c38f0888e90c4962cbf4ee3a0983077a7f0b9e77b1709b81"} Oct 30 21:39:09.477346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.477303 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" event={"ID":"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076","Type":"ContainerStarted","Data":"0f4462bc15f96623137f672889b3a338379a67e6bf66c118d20dd456009c77d8"} Oct 30 21:39:09.592915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.592856 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podStartSLOduration=1.592838812 podStartE2EDuration="1.592838812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:09.591780272 +0000 UTC m=+1339.725153470" watchObservedRunningTime="2026-10-30 21:39:09.592838812 +0000 UTC m=+1339.726212011" Oct 30 21:39:09.606890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.606862 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:09.986201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:09.986168 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e209c61-424e-4070-8358-6d4bc95ab185" path="/var/lib/kubelet/pods/0e209c61-424e-4070-8358-6d4bc95ab185/volumes" Oct 30 21:39:10.620716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:10.620684 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:39:10.621126 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:10.620744 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 21:39:11.615509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:11.615393 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:39:11.618587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:11.615466 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 21:39:12.481149 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:39:12.481115 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:39:12.613960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:12.613924 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:39:12.614238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:12.614209 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 21:39:13.615514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:13.614969 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:39:13.615514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:13.615036 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 21:39:14.607515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:14.607309 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:39:14.607515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:14.607373 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 21:39:15.516513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:15.515602 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:15.609318 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:15.609273 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:39:15.610076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:15.609625 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 21:39:15.646517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:15.644609 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:15.663516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:15.661639 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:15.820532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:15.818608 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:15.847674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:15.847613 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:15.928686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:15.928627 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.026982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.026754 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:39:16.132578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.129013 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.365511 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.358670 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.378528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.369231 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.495003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.492604 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.536996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.536939 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.537833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.537792 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:39:16.581310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.581271 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Oct 30 21:39:16.620178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.620133 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:39:16.634181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.634119 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 21:39:16.671747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.670308 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:39:16.671747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.670801 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:39:16.904670 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.904612 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.908511 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.905332 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 30 21:39:16.916514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.909190 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:39:16.920512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.916816 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:39:16.924514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.923891 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:39:16.924514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.924300 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:39:16.941570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.939774 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": context deadline exceeded" Oct 30 21:39:16.941570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.939885 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.941570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:16.940317 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:39:17.019509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.019064 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.043466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.041691 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.043466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.042154 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:39:17.165732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.165635 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:17.178664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.178634 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:39:17.188515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.187452 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:39:17.391523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.383542 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.539011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.538973 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 21:39:17.552814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.549588 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:17.552814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.549647 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.614508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.611440 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Oct 30 21:39:17.645524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.636343 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:39:17.647823 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.647452 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:39:17.648971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.648943 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 21:39:17.649193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.649165 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 21:39:17.671933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.671886 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.894255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.894168 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:39:17.927617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.927572 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.946054 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.946026 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:39:17.958774 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.958709 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:39:17.960138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:17.959710 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:39:18.000089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:18.000060 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:39:18.053193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:18.053159 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph"] Oct 30 21:39:18.056872 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:18.056838 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29a43e7f-22wph"] Oct 30 21:39:19.649657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:19.649621 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:19.650225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:19.649688 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:19.988699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:19.988610 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34621233-47c6-4dd8-9952-dc3978591643" path="/var/lib/kubelet/pods/34621233-47c6-4dd8-9952-dc3978591643/volumes" Oct 30 21:39:19.989088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:19.989058 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:19.989567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:19.989543 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:19.990304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:19.990285 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:39:20.854542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.854482 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c"] Oct 30 21:39:20.860292 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.860275 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:20.912593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.912553 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:20.913175 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.913141 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:20.913321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.913234 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:20.916244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.916220 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c"] Oct 30 21:39:20.943354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.943320 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:20.943545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.943379 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:20.943597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.943570 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwkzs\" (UniqueName: \"kubernetes.io/projected/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-kube-api-access-lwkzs\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:20.960527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:20.960483 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:39:21.044343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:21.044307 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:21.044573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:21.044374 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:21.044573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:21.044460 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkzs\" (UniqueName: \"kubernetes.io/projected/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-kube-api-access-lwkzs\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:21.045121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:21.045102 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:21.046951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:21.046934 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:21.101051 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:21.101017 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkzs\" (UniqueName: \"kubernetes.io/projected/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-kube-api-access-lwkzs\") pod \"maas-api-revoke-sub-428d85a5-fgq2c\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:21.173958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:21.173880 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:21.554441 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:21.554412 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f11d78e_9b98_4c72_bdb4_81ec4b31f3b1.slice/crio-323bbeaa44d70ec8fb553fb8757c585273734b150d6ca439f81927841c28e613 WatchSource:0}: Error finding container 323bbeaa44d70ec8fb553fb8757c585273734b150d6ca439f81927841c28e613: Status 404 returned error can't find the container with id 323bbeaa44d70ec8fb553fb8757c585273734b150d6ca439f81927841c28e613 Oct 30 21:39:21.554696 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:21.554680 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c"] Oct 30 21:39:22.045988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:22.045954 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8"] Oct 30 21:39:22.050206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:22.050179 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e90454-dndn8"] Oct 30 21:39:22.216816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:22.216783 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" event={"ID":"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1","Type":"ContainerStarted","Data":"623702df91ad30033987994d6dabcfeab8c6ed93cfa557cae575ddd7b1f9e361"} Oct 30 21:39:22.216998 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:22.216836 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" event={"ID":"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1","Type":"ContainerStarted","Data":"323bbeaa44d70ec8fb553fb8757c585273734b150d6ca439f81927841c28e613"} Oct 30 21:39:22.331646 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:22.331599 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" podStartSLOduration=2.331582685 podStartE2EDuration="2.331582685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:22.328843789 +0000 UTC m=+1352.462216990" watchObservedRunningTime="2026-10-30 21:39:22.331582685 +0000 UTC m=+1352.464955882" Oct 30 21:39:23.048278 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.048240 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc"] Oct 30 21:39:23.054021 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.053994 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6575b78a-rvgdc"] Oct 30 21:39:23.225705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.225669 4526 generic.go:356] "Generic (PLEG): container finished" podID="0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1" containerID="623702df91ad30033987994d6dabcfeab8c6ed93cfa557cae575ddd7b1f9e361" exitCode=0 Oct 30 21:39:23.225883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.225743 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" event={"ID":"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1","Type":"ContainerDied","Data":"623702df91ad30033987994d6dabcfeab8c6ed93cfa557cae575ddd7b1f9e361"} Oct 30 21:39:23.409671 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.409589 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552"] Oct 30 21:39:23.413997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.413974 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.434308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.434285 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw"] Oct 30 21:39:23.438263 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.438246 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.475871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.475845 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e78e-91e7-49d4-8078-406f90fb35e4-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.476038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.475890 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.476038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.475925 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.476038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.475975 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.476167 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.476133 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.476167 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.476165 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.476261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.476182 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgwtv\" (UniqueName: \"kubernetes.io/projected/75fb46d5-9526-4d36-a533-7974823eae7a-kube-api-access-rgwtv\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.476261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.476235 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.476339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.476297 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.476418 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.476406 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.476469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.476427 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.476555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.476535 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz42n\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-kube-api-access-zz42n\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.509374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.509343 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw"] Oct 30 21:39:23.514746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.514721 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552"] Oct 30 21:39:23.577319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577291 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e78e-91e7-49d4-8078-406f90fb35e4-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.577461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577326 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.577461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577356 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.577461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577372 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.577461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577432 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.577690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577546 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.577690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577609 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgwtv\" (UniqueName: \"kubernetes.io/projected/75fb46d5-9526-4d36-a533-7974823eae7a-kube-api-access-rgwtv\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.577690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577650 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.577855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577779 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.577855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577848 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.577989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.577969 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.578181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.578137 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.578181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.578004 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.578328 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.578220 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.578393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.578345 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e78e-91e7-49d4-8078-406f90fb35e4-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.578544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.578517 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz42n\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-kube-api-access-zz42n\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.578659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.578530 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.578869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.578849 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.580297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.580281 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.580680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.580653 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.580882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.580864 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.634853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.634826 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgwtv\" (UniqueName: \"kubernetes.io/projected/75fb46d5-9526-4d36-a533-7974823eae7a-kube-api-access-rgwtv\") pod \"maas-api-revoke-sub-cf3a3ff6-dbqrw\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.635442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.635413 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.635555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.635413 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz42n\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-kube-api-access-zz42n\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.727475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.727409 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:23.751394 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.751375 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:23.985749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.985677 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ceda648-9305-4c3c-b6ab-613e87c1fa57" path="/var/lib/kubelet/pods/3ceda648-9305-4c3c-b6ab-613e87c1fa57/volumes" Oct 30 21:39:23.986073 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:23.986060 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb90b91-6a9c-4822-a44d-31ab10ec0b8f" path="/var/lib/kubelet/pods/ffb90b91-6a9c-4822-a44d-31ab10ec0b8f/volumes" Oct 30 21:39:24.172793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.172768 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw"] Oct 30 21:39:24.173698 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:24.173674 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fb46d5_9526_4d36_a533_7974823eae7a.slice/crio-6fea8d7fd505122463bc94057be4b29f0f2243ebfa4ce78f75e9082f4c33d1d2 WatchSource:0}: Error finding container 6fea8d7fd505122463bc94057be4b29f0f2243ebfa4ce78f75e9082f4c33d1d2: Status 404 returned error can't find the container with id 6fea8d7fd505122463bc94057be4b29f0f2243ebfa4ce78f75e9082f4c33d1d2 Oct 30 21:39:24.174970 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:24.174945 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad8e78e_91e7_49d4_8078_406f90fb35e4.slice/crio-f7a9edeb04dfab67f2db3713c93c441574f04e4cba83b3d58d4c488295aecd9f WatchSource:0}: Error finding container f7a9edeb04dfab67f2db3713c93c441574f04e4cba83b3d58d4c488295aecd9f: Status 404 returned error can't find the container with id f7a9edeb04dfab67f2db3713c93c441574f04e4cba83b3d58d4c488295aecd9f Oct 30 21:39:24.177403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.177381 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552"] Oct 30 21:39:24.177506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.177416 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:24.177565 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.177482 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:24.177601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.177590 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:24.237713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.237685 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" event={"ID":"bad8e78e-91e7-49d4-8078-406f90fb35e4","Type":"ContainerStarted","Data":"f7a9edeb04dfab67f2db3713c93c441574f04e4cba83b3d58d4c488295aecd9f"} Oct 30 21:39:24.239037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.239011 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" event={"ID":"75fb46d5-9526-4d36-a533-7974823eae7a","Type":"ContainerStarted","Data":"6fea8d7fd505122463bc94057be4b29f0f2243ebfa4ce78f75e9082f4c33d1d2"} Oct 30 21:39:24.413205 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.413184 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:24.491375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.491298 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-xks-ca\") pod \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " Oct 30 21:39:24.491550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.491407 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-kube-api-access-lwkzs\" (UniqueName: \"kubernetes.io/projected/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-kube-api-access-lwkzs\") pod \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " Oct 30 21:39:24.491550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.491509 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-openshift-ca\") pod \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\" (UID: \"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1\") " Oct 30 21:39:24.491900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.491871 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-openshift-ca" pod "0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1" (UID: "0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:24.492316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.492298 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.493960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.493926 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-xks-ca" pod "0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1" (UID: "0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:24.494192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.494173 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-kube-api-access-lwkzs" pod "0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1" (UID: "0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1"). InnerVolumeSpecName "kube-api-access-lwkzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:24.593442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.593412 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:24.593569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:24.593449 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwkzs\" (UniqueName: \"kubernetes.io/projected/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1-kube-api-access-lwkzs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:25.248707 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.248679 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" Oct 30 21:39:25.249129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.248706 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c" event={"ID":"0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1","Type":"ContainerDied","Data":"323bbeaa44d70ec8fb553fb8757c585273734b150d6ca439f81927841c28e613"} Oct 30 21:39:25.249129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.248759 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="323bbeaa44d70ec8fb553fb8757c585273734b150d6ca439f81927841c28e613" Oct 30 21:39:25.250638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.250613 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" event={"ID":"bad8e78e-91e7-49d4-8078-406f90fb35e4","Type":"ContainerStarted","Data":"9f392e41777bec010e21aa99057cb85592f57e64e06d7f17389b943a57396dce"} Oct 30 21:39:25.252377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.252352 4526 generic.go:356] "Generic (PLEG): container finished" podID="75fb46d5-9526-4d36-a533-7974823eae7a" containerID="a4fe20f8f878e05dfd06c8d1173e4be10eb122ca497370ec0d5345c7234daf30" exitCode=0 Oct 30 21:39:25.252525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.252435 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" event={"ID":"75fb46d5-9526-4d36-a533-7974823eae7a","Type":"ContainerDied","Data":"a4fe20f8f878e05dfd06c8d1173e4be10eb122ca497370ec0d5345c7234daf30"} Oct 30 21:39:25.420041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.419985 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podStartSLOduration=2.419970136 podStartE2EDuration="2.419970136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:25.419686526 +0000 UTC m=+1355.553059735" watchObservedRunningTime="2026-10-30 21:39:25.419970136 +0000 UTC m=+1355.553343333" Oct 30 21:39:25.728506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.728447 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:25.730232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.730206 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:39:25.730337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:25.730265 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:39:26.458982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.458956 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:26.536240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.536216 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-xks-ca\") pod \"75fb46d5-9526-4d36-a533-7974823eae7a\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " Oct 30 21:39:26.536403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.536262 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-openshift-ca\") pod \"75fb46d5-9526-4d36-a533-7974823eae7a\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " Oct 30 21:39:26.536403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.536393 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75fb46d5-9526-4d36-a533-7974823eae7a-kube-api-access-rgwtv\" (UniqueName: \"kubernetes.io/projected/75fb46d5-9526-4d36-a533-7974823eae7a-kube-api-access-rgwtv\") pod \"75fb46d5-9526-4d36-a533-7974823eae7a\" (UID: \"75fb46d5-9526-4d36-a533-7974823eae7a\") " Oct 30 21:39:26.536662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.536636 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-openshift-ca" pod "75fb46d5-9526-4d36-a533-7974823eae7a" (UID: "75fb46d5-9526-4d36-a533-7974823eae7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:26.537000 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.536985 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:26.538559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.538532 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-xks-ca" pod "75fb46d5-9526-4d36-a533-7974823eae7a" (UID: "75fb46d5-9526-4d36-a533-7974823eae7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:26.538656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.538619 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fb46d5-9526-4d36-a533-7974823eae7a-kube-api-access-rgwtv" pod "75fb46d5-9526-4d36-a533-7974823eae7a" (UID: "75fb46d5-9526-4d36-a533-7974823eae7a"). InnerVolumeSpecName "kube-api-access-rgwtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:26.638042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.638008 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgwtv\" (UniqueName: \"kubernetes.io/projected/75fb46d5-9526-4d36-a533-7974823eae7a-kube-api-access-rgwtv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:26.638042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.638044 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75fb46d5-9526-4d36-a533-7974823eae7a-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:26.728063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.728026 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:39:26.728220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:26.728086 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:39:27.270838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:27.270811 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" Oct 30 21:39:27.270838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:27.270809 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw" event={"ID":"75fb46d5-9526-4d36-a533-7974823eae7a","Type":"ContainerDied","Data":"6fea8d7fd505122463bc94057be4b29f0f2243ebfa4ce78f75e9082f4c33d1d2"} Oct 30 21:39:27.271067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:27.270864 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fea8d7fd505122463bc94057be4b29f0f2243ebfa4ce78f75e9082f4c33d1d2" Oct 30 21:39:27.728251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:27.728219 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:39:27.728712 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:27.728277 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:39:28.521124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.521093 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849"] Oct 30 21:39:28.521664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.521650 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1" containerName="revoke-keys" Oct 30 21:39:28.521717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.521666 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1" containerName="revoke-keys" Oct 30 21:39:28.521717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.521708 4526 cpu_manager.go:414] "containerMap: removing container" podUID="75fb46d5-9526-4d36-a533-7974823eae7a" containerName="revoke-keys" Oct 30 21:39:28.521717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.521715 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75fb46d5-9526-4d36-a533-7974823eae7a" containerName="revoke-keys" Oct 30 21:39:28.521810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.521796 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1" containerName="revoke-keys" Oct 30 21:39:28.521849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.521811 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="75fb46d5-9526-4d36-a533-7974823eae7a" containerName="revoke-keys" Oct 30 21:39:28.525623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.525598 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.562241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.562205 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.562410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.562276 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4f6b\" (UniqueName: \"kubernetes.io/projected/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-kube-api-access-h4f6b\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.562524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.562429 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.563269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.563244 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849"] Oct 30 21:39:28.617895 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.617863 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:28.618322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.618153 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:28.618572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.618279 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:28.626609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.626587 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:39:28.663956 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.663925 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4f6b\" (UniqueName: \"kubernetes.io/projected/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-kube-api-access-h4f6b\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.664115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.663991 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.664197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.664115 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.664610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.664588 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.666608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.666585 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.721391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.721357 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4f6b\" (UniqueName: \"kubernetes.io/projected/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-kube-api-access-h4f6b\") pod \"maas-api-revoke-sub-348cabb3-b2849\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:28.728455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.728428 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:39:28.728821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.728500 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:39:28.839321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:28.839290 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:29.228316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:29.228284 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849"] Oct 30 21:39:29.228623 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:29.228594 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaebcde1_c1ae_4e37_a475_2a48cd0cebdb.slice/crio-d01d8d6074ae818a84c1cbe941a31a382c67b1bb210aa7bcbfd085b24113912a WatchSource:0}: Error finding container d01d8d6074ae818a84c1cbe941a31a382c67b1bb210aa7bcbfd085b24113912a: Status 404 returned error can't find the container with id d01d8d6074ae818a84c1cbe941a31a382c67b1bb210aa7bcbfd085b24113912a Oct 30 21:39:29.288145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:29.288121 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" event={"ID":"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb","Type":"ContainerStarted","Data":"d01d8d6074ae818a84c1cbe941a31a382c67b1bb210aa7bcbfd085b24113912a"} Oct 30 21:39:29.728408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:29.728371 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:39:29.728600 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:29.728447 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:39:30.301208 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:30.301171 4526 generic.go:356] "Generic (PLEG): container finished" podID="eaebcde1-c1ae-4e37-a475-2a48cd0cebdb" containerID="f8429e299789bb324dc918ac975b90d8bc9923903c01a1877925be4588fd30f2" exitCode=0 Oct 30 21:39:30.301437 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:30.301261 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" event={"ID":"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb","Type":"ContainerDied","Data":"f8429e299789bb324dc918ac975b90d8bc9923903c01a1877925be4588fd30f2"} Oct 30 21:39:30.728197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:30.728156 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:39:30.728374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:30.728237 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 21:39:31.499955 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.499927 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:31.604104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.604013 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-openshift-ca\") pod \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " Oct 30 21:39:31.604104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.604067 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-kube-api-access-h4f6b\" (UniqueName: \"kubernetes.io/projected/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-kube-api-access-h4f6b\") pod \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " Oct 30 21:39:31.604363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.604278 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-xks-ca\") pod \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\" (UID: \"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb\") " Oct 30 21:39:31.604436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.604357 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-openshift-ca" pod "eaebcde1-c1ae-4e37-a475-2a48cd0cebdb" (UID: "eaebcde1-c1ae-4e37-a475-2a48cd0cebdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:31.604712 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.604695 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:31.606552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.606527 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-kube-api-access-h4f6b" pod "eaebcde1-c1ae-4e37-a475-2a48cd0cebdb" (UID: "eaebcde1-c1ae-4e37-a475-2a48cd0cebdb"). InnerVolumeSpecName "kube-api-access-h4f6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:31.606651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.606562 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-xks-ca" pod "eaebcde1-c1ae-4e37-a475-2a48cd0cebdb" (UID: "eaebcde1-c1ae-4e37-a475-2a48cd0cebdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:31.705730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.705692 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:31.705730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.705720 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4f6b\" (UniqueName: \"kubernetes.io/projected/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb-kube-api-access-h4f6b\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:31.733052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.733018 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:31.733524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.733508 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:31.734274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:31.734256 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:39:32.319522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.319456 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" Oct 30 21:39:32.319775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.319483 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849" event={"ID":"eaebcde1-c1ae-4e37-a475-2a48cd0cebdb","Type":"ContainerDied","Data":"d01d8d6074ae818a84c1cbe941a31a382c67b1bb210aa7bcbfd085b24113912a"} Oct 30 21:39:32.319904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.319807 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d01d8d6074ae818a84c1cbe941a31a382c67b1bb210aa7bcbfd085b24113912a" Oct 30 21:39:32.477764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.477735 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf"] Oct 30 21:39:32.478325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.478312 4526 cpu_manager.go:414] "containerMap: removing container" podUID="eaebcde1-c1ae-4e37-a475-2a48cd0cebdb" containerName="revoke-keys" Oct 30 21:39:32.478366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.478328 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaebcde1-c1ae-4e37-a475-2a48cd0cebdb" containerName="revoke-keys" Oct 30 21:39:32.478419 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.478410 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaebcde1-c1ae-4e37-a475-2a48cd0cebdb" containerName="revoke-keys" Oct 30 21:39:32.482108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.482090 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.534557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.534323 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:32.535176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.534600 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:32.535176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.534733 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:32.535176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.534820 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:39:32.539066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.539045 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf"] Oct 30 21:39:32.618012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.617908 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.618012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.617962 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8m47\" (UniqueName: \"kubernetes.io/projected/1f351c93-77a7-4cbb-8dc5-d27470e54256-kube-api-access-n8m47\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.618207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.618068 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.719181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.719149 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.719375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.719189 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8m47\" (UniqueName: \"kubernetes.io/projected/1f351c93-77a7-4cbb-8dc5-d27470e54256-kube-api-access-n8m47\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.719375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.719238 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.719518 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.719479 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2"] Oct 30 21:39:32.720103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.720084 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.721820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.721795 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.724524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.724502 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.733890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.733869 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7"] Oct 30 21:39:32.737934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.737918 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.775257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.775221 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8m47\" (UniqueName: \"kubernetes.io/projected/1f351c93-77a7-4cbb-8dc5-d27470e54256-kube-api-access-n8m47\") pod \"maas-api-revoke-sub-3793273d-gtxxf\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.782747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.782722 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2"] Oct 30 21:39:32.798535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.795882 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:32.798535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.797542 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7"] Oct 30 21:39:32.823526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823476 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.823705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823560 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fdd38d-4425-45d1-a101-975dc6c631fb-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.823705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823592 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jnc\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-kube-api-access-p2jnc\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.823705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823656 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.823705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823703 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.823970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823782 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.823970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823819 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.823970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823857 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a1029-a761-46d7-a09a-5ab39fe6f94d-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.823970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823894 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.823970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823918 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.824221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.823989 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.824221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.824028 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.824221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.824051 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.824221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.824112 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.824221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.824163 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.824221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.824197 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.824551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.824240 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz48n\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-kube-api-access-fz48n\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.824551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.824288 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925122 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925170 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925186 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.925440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925217 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925246 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.925440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925275 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.925440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925314 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz48n\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-kube-api-access-fz48n\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.925440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925356 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925433 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925478 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fdd38d-4425-45d1-a101-975dc6c631fb-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925670 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.925887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925722 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925735 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jnc\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-kube-api-access-p2jnc\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.925887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925821 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.925887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925878 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.926214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925965 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.926214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.925967 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.926214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926004 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.926214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926043 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a1029-a761-46d7-a09a-5ab39fe6f94d-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.926214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926081 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.926214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926110 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.926214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926186 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fdd38d-4425-45d1-a101-975dc6c631fb-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.926214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926213 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.926662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926617 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.926799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926763 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.926887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926866 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.926980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.926957 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.927609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.927583 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a1029-a761-46d7-a09a-5ab39fe6f94d-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.928676 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.928643 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.928959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.928931 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.929180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.929162 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.929860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.929841 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.981787 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.981711 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.981978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.981790 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:32.981978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.981821 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz48n\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-kube-api-access-fz48n\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:32.982668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:32.982645 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jnc\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-kube-api-access-p2jnc\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:33.050098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.050064 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:33.052812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.052782 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d"] Oct 30 21:39:33.056933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.056912 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12facb17-gzp9d"] Oct 30 21:39:33.057144 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.057125 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:33.198531 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.198500 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf"] Oct 30 21:39:33.199305 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:33.199273 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f351c93_77a7_4cbb_8dc5_d27470e54256.slice/crio-2e27423d11bf297bec88d84db5c3a9ff56e832e3e966eb480d45c1d4b8635c64 WatchSource:0}: Error finding container 2e27423d11bf297bec88d84db5c3a9ff56e832e3e966eb480d45c1d4b8635c64: Status 404 returned error can't find the container with id 2e27423d11bf297bec88d84db5c3a9ff56e832e3e966eb480d45c1d4b8635c64 Oct 30 21:39:33.313525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.313474 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg"] Oct 30 21:39:33.319211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.319188 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.334803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.334771 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" event={"ID":"1f351c93-77a7-4cbb-8dc5-d27470e54256","Type":"ContainerStarted","Data":"2e27423d11bf297bec88d84db5c3a9ff56e832e3e966eb480d45c1d4b8635c64"} Oct 30 21:39:33.377361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.376087 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg"] Oct 30 21:39:33.433770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.433729 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.433979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.433822 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grl4s\" (UniqueName: \"kubernetes.io/projected/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-kube-api-access-grl4s\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.433979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.433883 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.441884 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:33.441855 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fdd38d_4425_45d1_a101_975dc6c631fb.slice/crio-3065877b1e484348ca57fa82825341c226979fd03c5e4f8f93b235980cff1c24 WatchSource:0}: Error finding container 3065877b1e484348ca57fa82825341c226979fd03c5e4f8f93b235980cff1c24: Status 404 returned error can't find the container with id 3065877b1e484348ca57fa82825341c226979fd03c5e4f8f93b235980cff1c24 Oct 30 21:39:33.442052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.442031 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2"] Oct 30 21:39:33.444629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.444590 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:33.444705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.444658 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:33.444705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.444687 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:33.493359 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:33.493330 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054a1029_a761_46d7_a09a_5ab39fe6f94d.slice/crio-be9ac6e3d1d82ba153536a3145326c819dec555f81bc467587f729dad4adeacb WatchSource:0}: Error finding container be9ac6e3d1d82ba153536a3145326c819dec555f81bc467587f729dad4adeacb: Status 404 returned error can't find the container with id be9ac6e3d1d82ba153536a3145326c819dec555f81bc467587f729dad4adeacb Oct 30 21:39:33.493544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.493523 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7"] Oct 30 21:39:33.496679 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.496646 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:33.496765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.496730 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:33.496808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.496772 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:33.535135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.535105 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.535564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.535175 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grl4s\" (UniqueName: \"kubernetes.io/projected/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-kube-api-access-grl4s\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.535564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.535206 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.536106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.536076 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.537842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.537818 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.591877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.591845 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grl4s\" (UniqueName: \"kubernetes.io/projected/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-kube-api-access-grl4s\") pod \"maas-api-revoke-sub-591b8451-w9lwg\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.638210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.638175 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:33.986629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:33.986536 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0302e6-03e3-418f-b6e7-43a485ec5537" path="/var/lib/kubelet/pods/3d0302e6-03e3-418f-b6e7-43a485ec5537/volumes" Oct 30 21:39:34.043516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.032191 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg"] Oct 30 21:39:34.044742 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:34.044713 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b96cf3f_18c8_45f5_a381_ab923fcd5f7b.slice/crio-a05fa002428f6fc786b84a25de447ec77c1290df48e4a2c92fb4d9caab9a4387 WatchSource:0}: Error finding container a05fa002428f6fc786b84a25de447ec77c1290df48e4a2c92fb4d9caab9a4387: Status 404 returned error can't find the container with id a05fa002428f6fc786b84a25de447ec77c1290df48e4a2c92fb4d9caab9a4387 Oct 30 21:39:34.352195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.352133 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" event={"ID":"37fdd38d-4425-45d1-a101-975dc6c631fb","Type":"ContainerStarted","Data":"79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec"} Oct 30 21:39:34.352390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.352204 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" event={"ID":"37fdd38d-4425-45d1-a101-975dc6c631fb","Type":"ContainerStarted","Data":"3065877b1e484348ca57fa82825341c226979fd03c5e4f8f93b235980cff1c24"} Oct 30 21:39:34.353513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.353470 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" event={"ID":"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b","Type":"ContainerStarted","Data":"a05fa002428f6fc786b84a25de447ec77c1290df48e4a2c92fb4d9caab9a4387"} Oct 30 21:39:34.355237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.355208 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" event={"ID":"054a1029-a761-46d7-a09a-5ab39fe6f94d","Type":"ContainerStarted","Data":"69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee"} Oct 30 21:39:34.355315 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.355265 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" event={"ID":"054a1029-a761-46d7-a09a-5ab39fe6f94d","Type":"ContainerStarted","Data":"be9ac6e3d1d82ba153536a3145326c819dec555f81bc467587f729dad4adeacb"} Oct 30 21:39:34.356898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.356871 4526 generic.go:356] "Generic (PLEG): container finished" podID="1f351c93-77a7-4cbb-8dc5-d27470e54256" containerID="9baa1a0a451002f866f90d03e3b5e23d82a0a2d6ccbe20ff3430e44fe59d6155" exitCode=0 Oct 30 21:39:34.357010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.356945 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" event={"ID":"1f351c93-77a7-4cbb-8dc5-d27470e54256","Type":"ContainerDied","Data":"9baa1a0a451002f866f90d03e3b5e23d82a0a2d6ccbe20ff3430e44fe59d6155"} Oct 30 21:39:34.514573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.514426 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podStartSLOduration=2.514404042 podStartE2EDuration="2.514404042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:34.490790207 +0000 UTC m=+1364.624163407" watchObservedRunningTime="2026-10-30 21:39:34.514404042 +0000 UTC m=+1364.647777243" Oct 30 21:39:34.752957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:34.751736 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podStartSLOduration=2.751719669 podStartE2EDuration="2.751719669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:34.748938433 +0000 UTC m=+1364.882311634" watchObservedRunningTime="2026-10-30 21:39:34.751719669 +0000 UTC m=+1364.885092868" Oct 30 21:39:35.051693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.051342 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:35.052743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.052720 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:39:35.052819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.052764 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 21:39:35.058570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.058124 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:35.059084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.059067 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:39:35.059150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.059100 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 21:39:35.366192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.366104 4526 generic.go:356] "Generic (PLEG): container finished" podID="5b96cf3f-18c8-45f5-a381-ab923fcd5f7b" containerID="4ef51a2f239ba57464a23424515cb7c2a8b30d9fa5494cfd5731ba8a768050f3" exitCode=0 Oct 30 21:39:35.366429 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.366404 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" event={"ID":"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b","Type":"ContainerDied","Data":"4ef51a2f239ba57464a23424515cb7c2a8b30d9fa5494cfd5731ba8a768050f3"} Oct 30 21:39:35.680899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.680873 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:35.773443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.773412 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f351c93-77a7-4cbb-8dc5-d27470e54256-kube-api-access-n8m47\" (UniqueName: \"kubernetes.io/projected/1f351c93-77a7-4cbb-8dc5-d27470e54256-kube-api-access-n8m47\") pod \"1f351c93-77a7-4cbb-8dc5-d27470e54256\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " Oct 30 21:39:35.773976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.773601 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-xks-ca\") pod \"1f351c93-77a7-4cbb-8dc5-d27470e54256\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " Oct 30 21:39:35.773976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.773709 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-openshift-ca\") pod \"1f351c93-77a7-4cbb-8dc5-d27470e54256\" (UID: \"1f351c93-77a7-4cbb-8dc5-d27470e54256\") " Oct 30 21:39:35.774686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.774654 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-openshift-ca" pod "1f351c93-77a7-4cbb-8dc5-d27470e54256" (UID: "1f351c93-77a7-4cbb-8dc5-d27470e54256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:35.776804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.776778 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-xks-ca" pod "1f351c93-77a7-4cbb-8dc5-d27470e54256" (UID: "1f351c93-77a7-4cbb-8dc5-d27470e54256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:35.776958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.776819 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f351c93-77a7-4cbb-8dc5-d27470e54256-kube-api-access-n8m47" pod "1f351c93-77a7-4cbb-8dc5-d27470e54256" (UID: "1f351c93-77a7-4cbb-8dc5-d27470e54256"). InnerVolumeSpecName "kube-api-access-n8m47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:35.875581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.875540 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8m47\" (UniqueName: \"kubernetes.io/projected/1f351c93-77a7-4cbb-8dc5-d27470e54256-kube-api-access-n8m47\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:35.875581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.875574 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:35.875581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:35.875589 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f351c93-77a7-4cbb-8dc5-d27470e54256-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:36.051106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.051073 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:39:36.051266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.051133 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 21:39:36.057754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.057721 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:39:36.057928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.057779 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 21:39:36.376730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.376649 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" Oct 30 21:39:36.376730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.376671 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf" event={"ID":"1f351c93-77a7-4cbb-8dc5-d27470e54256","Type":"ContainerDied","Data":"2e27423d11bf297bec88d84db5c3a9ff56e832e3e966eb480d45c1d4b8635c64"} Oct 30 21:39:36.376730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.376722 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e27423d11bf297bec88d84db5c3a9ff56e832e3e966eb480d45c1d4b8635c64" Oct 30 21:39:36.677777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.677753 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:36.787540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.787467 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-openshift-ca\") pod \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " Oct 30 21:39:36.787942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.787562 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-kube-api-access-grl4s\" (UniqueName: \"kubernetes.io/projected/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-kube-api-access-grl4s\") pod \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " Oct 30 21:39:36.787942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.787590 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-xks-ca\") pod \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\" (UID: \"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b\") " Oct 30 21:39:36.788036 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.787983 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-openshift-ca" pod "5b96cf3f-18c8-45f5-a381-ab923fcd5f7b" (UID: "5b96cf3f-18c8-45f5-a381-ab923fcd5f7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:36.788100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.788088 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:36.789918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.789886 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-kube-api-access-grl4s" pod "5b96cf3f-18c8-45f5-a381-ab923fcd5f7b" (UID: "5b96cf3f-18c8-45f5-a381-ab923fcd5f7b"). InnerVolumeSpecName "kube-api-access-grl4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:36.790224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.790205 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-xks-ca" pod "5b96cf3f-18c8-45f5-a381-ab923fcd5f7b" (UID: "5b96cf3f-18c8-45f5-a381-ab923fcd5f7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:36.889319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.889284 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grl4s\" (UniqueName: \"kubernetes.io/projected/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-kube-api-access-grl4s\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:36.889319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:36.889317 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:37.051383 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:37.051337 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:39:37.051585 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:37.051396 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 21:39:37.057734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:37.057709 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:39:37.057823 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:37.057757 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 21:39:37.387775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:37.386836 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" event={"ID":"5b96cf3f-18c8-45f5-a381-ab923fcd5f7b","Type":"ContainerDied","Data":"a05fa002428f6fc786b84a25de447ec77c1290df48e4a2c92fb4d9caab9a4387"} Oct 30 21:39:37.387775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:37.386884 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a05fa002428f6fc786b84a25de447ec77c1290df48e4a2c92fb4d9caab9a4387" Oct 30 21:39:37.387775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:37.386936 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg" Oct 30 21:39:38.051211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.051180 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:39:38.051706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.051680 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 21:39:38.057939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.057912 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:39:38.058075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.057948 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 21:39:38.148077 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.147757 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27"] Oct 30 21:39:38.148756 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.148724 4526 cpu_manager.go:414] "containerMap: removing container" podUID="5b96cf3f-18c8-45f5-a381-ab923fcd5f7b" containerName="revoke-keys" Oct 30 21:39:38.148756 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.148750 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b96cf3f-18c8-45f5-a381-ab923fcd5f7b" containerName="revoke-keys" Oct 30 21:39:38.148935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.148818 4526 cpu_manager.go:414] "containerMap: removing container" podUID="1f351c93-77a7-4cbb-8dc5-d27470e54256" containerName="revoke-keys" Oct 30 21:39:38.148935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.148828 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f351c93-77a7-4cbb-8dc5-d27470e54256" containerName="revoke-keys" Oct 30 21:39:38.149012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.148960 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b96cf3f-18c8-45f5-a381-ab923fcd5f7b" containerName="revoke-keys" Oct 30 21:39:38.149012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.148981 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f351c93-77a7-4cbb-8dc5-d27470e54256" containerName="revoke-keys" Oct 30 21:39:38.152921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.152904 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.214704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.214672 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9796f1-openshift-default-dockercfg-bzdq6\"" Oct 30 21:39:38.242842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.242805 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istiod-ca-cert\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.243435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.243414 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-credential-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.243655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.243637 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-podinfo\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.243831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.243817 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-envoy\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.243969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.243955 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-certs\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.244065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.244053 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-token\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.244214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.244198 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.244312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.244299 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-data\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.244465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.244448 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7g6c\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-kube-api-access-b7g6c\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.268096 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.268062 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27"] Oct 30 21:39:38.350610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.350513 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istiod-ca-cert\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.350919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.350896 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-credential-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.351033 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.351020 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-podinfo\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.351202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.351188 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-envoy\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.352089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.352065 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-certs\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.352252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.352238 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-token\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.352403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.352379 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.352745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.352708 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-credential-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.352901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.352725 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-data\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.353076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.353057 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7g6c\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-kube-api-access-b7g6c\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.353176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.353153 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-certs\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.353234 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.352674 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istiod-ca-cert\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.353394 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.352957 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-data\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.353544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.353405 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.355346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.355321 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-envoy\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.356800 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.356782 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-podinfo\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.457869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.457837 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-token\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.459269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.459244 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7g6c\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-kube-api-access-b7g6c\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.467089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.467066 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:38.846524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.844043 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8"] Oct 30 21:39:38.850996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.850968 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.957104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.956949 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8"] Oct 30 21:39:38.966643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.966606 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac25a73-df68-4c82-989d-2dd28e379484-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.966826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.966693 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac25a73-df68-4c82-989d-2dd28e379484-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.966910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.966876 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.966968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.966903 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.966968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.966930 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6kv\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-kube-api-access-9g6kv\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.967068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.966971 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.967068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.967003 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.967068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.967043 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.967220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.967120 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:38.992479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:38.992279 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27"] Oct 30 21:39:38.992728 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:38.992708 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f64983_5fc2_4bcf_a97c_71de7483afa5.slice/crio-cd64c799199ef473f71537d8f21a070d96bfa1da9e5070dac5f70fe08030bbce WatchSource:0}: Error finding container cd64c799199ef473f71537d8f21a070d96bfa1da9e5070dac5f70fe08030bbce: Status 404 returned error can't find the container with id cd64c799199ef473f71537d8f21a070d96bfa1da9e5070dac5f70fe08030bbce Oct 30 21:39:39.049201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.049165 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.049340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.049243 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.049340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.049274 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.050453 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.050432 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:39:39.050563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.050472 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 21:39:39.057678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.057655 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:39:39.058010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.057692 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 21:39:39.068470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068448 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.068562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068533 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac25a73-df68-4c82-989d-2dd28e379484-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.068595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068570 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac25a73-df68-4c82-989d-2dd28e379484-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.068689 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068678 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.068728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068696 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.068728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068712 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6kv\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-kube-api-access-9g6kv\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.068788 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068739 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.068788 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068757 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.068788 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.068780 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.069293 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.069269 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.069826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.069805 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac25a73-df68-4c82-989d-2dd28e379484-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.069904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.069816 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.069904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.069871 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.070083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.070066 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.071551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.071529 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.074185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.074166 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac25a73-df68-4c82-989d-2dd28e379484-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.149141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.149115 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.152172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.152146 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6kv\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-kube-api-access-9g6kv\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.165252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.165221 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:39.470806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.470712 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" event={"ID":"e8f64983-5fc2-4bcf-a97c-71de7483afa5","Type":"ContainerStarted","Data":"f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d"} Oct 30 21:39:39.470806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.470776 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" event={"ID":"e8f64983-5fc2-4bcf-a97c-71de7483afa5","Type":"ContainerStarted","Data":"cd64c799199ef473f71537d8f21a070d96bfa1da9e5070dac5f70fe08030bbce"} Oct 30 21:39:39.633515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.629926 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podStartSLOduration=1.6299037410000001 podStartE2EDuration="1.629903741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:39.620989313 +0000 UTC m=+1369.754362514" watchObservedRunningTime="2026-10-30 21:39:39.629903741 +0000 UTC m=+1369.763276941" Oct 30 21:39:39.735002 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.734967 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8"] Oct 30 21:39:39.736174 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:39.736143 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac25a73_df68_4c82_989d_2dd28e379484.slice/crio-16ab9dae772c8d72271cc3cdf54293fbac8ac3438e37596588ed8d07877ef21c WatchSource:0}: Error finding container 16ab9dae772c8d72271cc3cdf54293fbac8ac3438e37596588ed8d07877ef21c: Status 404 returned error can't find the container with id 16ab9dae772c8d72271cc3cdf54293fbac8ac3438e37596588ed8d07877ef21c Oct 30 21:39:39.740029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.739983 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.740146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.740087 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.740146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:39.740133 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:40.050977 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.050938 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 21:39:40.051301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.051268 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 21:39:40.057627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.057593 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 21:39:40.057731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.057649 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 21:39:40.471574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.467525 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:40.471574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.469703 4526 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:39:40.471574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.469757 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:39:40.489625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.489588 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" event={"ID":"dac25a73-df68-4c82-989d-2dd28e379484","Type":"ContainerStarted","Data":"58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e"} Oct 30 21:39:40.489776 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.489660 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" event={"ID":"dac25a73-df68-4c82-989d-2dd28e379484","Type":"ContainerStarted","Data":"16ab9dae772c8d72271cc3cdf54293fbac8ac3438e37596588ed8d07877ef21c"} Oct 30 21:39:40.630514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.629775 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podStartSLOduration=2.629750669 podStartE2EDuration="2.629750669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:40.610872495 +0000 UTC m=+1370.744245698" watchObservedRunningTime="2026-10-30 21:39:40.629750669 +0000 UTC m=+1370.763123868" Oct 30 21:39:40.776394 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.776291 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw"] Oct 30 21:39:40.782113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.781716 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:40.842515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.841977 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:40.846515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.842945 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:40.846515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.843019 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:40.846515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.843166 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:39:40.864514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.858773 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw"] Oct 30 21:39:40.945526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.922542 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:40.945526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.922643 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24m4\" (UniqueName: \"kubernetes.io/projected/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-kube-api-access-p24m4\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:40.945526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:40.922953 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:41.025515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.024546 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:41.025515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.024671 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:41.025515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.024732 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24m4\" (UniqueName: \"kubernetes.io/projected/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-kube-api-access-p24m4\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:41.045515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.042220 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:41.070516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.051778 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:41.087516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.087188 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24m4\" (UniqueName: \"kubernetes.io/projected/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-kube-api-access-p24m4\") pod \"maas-api-revoke-sub-f9597406-9drzw\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:41.126515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.113812 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:41.188599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.188541 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:41.191697 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.191655 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:39:41.191886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.191845 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:39:41.472521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.471752 4526 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:39:41.472521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.471832 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:39:41.700967 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.700872 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:41.707528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.702785 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:41.720236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.718784 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:39:41.727514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.723409 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt"] Oct 30 21:39:41.871690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.863725 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:41.876691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.876662 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:41.877970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.877946 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:41.881111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.881085 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:39:41.884890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:41.882784 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt"] Oct 30 21:39:42.060143 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.060111 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.060814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.060268 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgjr\" (UniqueName: \"kubernetes.io/projected/674fa73b-55ac-4158-a819-83483a19fb32-kube-api-access-ztgjr\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.060814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.060298 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.145578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.145541 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw"] Oct 30 21:39:42.150015 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:42.149988 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4215fa7d_c8cc_47f3_a8f8_b7d80b64418d.slice/crio-85f3b4f485eca2efa8939dd31aa4b3de23cf6f3a2006efc505da5d1e4a2dc0aa WatchSource:0}: Error finding container 85f3b4f485eca2efa8939dd31aa4b3de23cf6f3a2006efc505da5d1e4a2dc0aa: Status 404 returned error can't find the container with id 85f3b4f485eca2efa8939dd31aa4b3de23cf6f3a2006efc505da5d1e4a2dc0aa Oct 30 21:39:42.161345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.161320 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgjr\" (UniqueName: \"kubernetes.io/projected/674fa73b-55ac-4158-a819-83483a19fb32-kube-api-access-ztgjr\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.161507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.161363 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.161507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.161430 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.162064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.162039 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.164237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.164220 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.166457 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.166435 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:39:42.166556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.166479 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:39:42.253057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.244320 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgjr\" (UniqueName: \"kubernetes.io/projected/674fa73b-55ac-4158-a819-83483a19fb32-kube-api-access-ztgjr\") pod \"maas-api-revoke-sub-f9f3038e-rk8bt\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.468070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.468040 4526 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:39:42.468228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.468095 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:39:42.550167 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:39:42.550128 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:39:42.550167 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.550158 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:42.553427 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.553400 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" event={"ID":"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d","Type":"ContainerStarted","Data":"85f3b4f485eca2efa8939dd31aa4b3de23cf6f3a2006efc505da5d1e4a2dc0aa"} Oct 30 21:39:42.988428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:42.988388 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt"] Oct 30 21:39:42.988583 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:42.988561 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674fa73b_55ac_4158_a819_83483a19fb32.slice/crio-c9d11e41025d0cdf59f7e2a5f0e70f9313bb9c0dac739bd75fdceecae10191ca WatchSource:0}: Error finding container c9d11e41025d0cdf59f7e2a5f0e70f9313bb9c0dac739bd75fdceecae10191ca: Status 404 returned error can't find the container with id c9d11e41025d0cdf59f7e2a5f0e70f9313bb9c0dac739bd75fdceecae10191ca Oct 30 21:39:43.166107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.166069 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:39:43.166637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.166130 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:39:43.268273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.268242 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw"] Oct 30 21:39:43.274174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.274148 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.363277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.363232 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw"] Oct 30 21:39:43.375414 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.375056 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.375414 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.375155 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qg8\" (UniqueName: \"kubernetes.io/projected/0a3ee4a0-d451-4b8b-8586-da08c01d8321-kube-api-access-j6qg8\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.375414 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.375188 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.468051 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.468029 4526 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:39:43.468262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.468242 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:39:43.476338 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.476307 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qg8\" (UniqueName: \"kubernetes.io/projected/0a3ee4a0-d451-4b8b-8586-da08c01d8321-kube-api-access-j6qg8\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.476532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.476350 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.476968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.476943 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.477337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.477281 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.484003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.483977 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.561955 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.561890 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qg8\" (UniqueName: \"kubernetes.io/projected/0a3ee4a0-d451-4b8b-8586-da08c01d8321-kube-api-access-j6qg8\") pod \"maas-api-revoke-sub-85dc19fa-56zzw\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:43.571420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.571352 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" event={"ID":"674fa73b-55ac-4158-a819-83483a19fb32","Type":"ContainerStarted","Data":"c9d11e41025d0cdf59f7e2a5f0e70f9313bb9c0dac739bd75fdceecae10191ca"} Oct 30 21:39:43.573285 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.573157 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" event={"ID":"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d","Type":"ContainerStarted","Data":"1f65ac5708470d9890d9c94caa0959b18199835b3e8caa21204dc6bbfcc00ca1"} Oct 30 21:39:43.588533 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:43.588394 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:44.060628 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:44.060589 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3ee4a0_d451_4b8b_8586_da08c01d8321.slice/crio-a33794bd67df4b6569d5ac6bf871cc82e0b2cf536e044b8fdf590202678e02f4 WatchSource:0}: Error finding container a33794bd67df4b6569d5ac6bf871cc82e0b2cf536e044b8fdf590202678e02f4: Status 404 returned error can't find the container with id a33794bd67df4b6569d5ac6bf871cc82e0b2cf536e044b8fdf590202678e02f4 Oct 30 21:39:44.063636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.063604 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw"] Oct 30 21:39:44.153254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.153209 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh"] Oct 30 21:39:44.159085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.159064 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.167516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.167277 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:39:44.167516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.167328 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:39:44.181073 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.181039 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh"] Oct 30 21:39:44.197513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.193012 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44vp\" (UniqueName: \"kubernetes.io/projected/31bb8bee-a24f-479b-8331-bca0f146a40d-kube-api-access-h44vp\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.197513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.193237 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.197513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.193293 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.337513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.321290 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.337857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.337828 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.338152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.338137 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44vp\" (UniqueName: \"kubernetes.io/projected/31bb8bee-a24f-479b-8331-bca0f146a40d-kube-api-access-h44vp\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.349213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.349180 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.357504 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.356787 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.441692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.441355 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44vp\" (UniqueName: \"kubernetes.io/projected/31bb8bee-a24f-479b-8331-bca0f146a40d-kube-api-access-h44vp\") pod \"maas-api-revoke-sub-91f93ecb-v5rlh\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.467576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.467542 4526 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:39:44.467754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.467604 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:39:44.480211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.480176 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:44.584429 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.584345 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" event={"ID":"0a3ee4a0-d451-4b8b-8586-da08c01d8321","Type":"ContainerStarted","Data":"a33794bd67df4b6569d5ac6bf871cc82e0b2cf536e044b8fdf590202678e02f4"} Oct 30 21:39:44.586931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.586902 4526 generic.go:356] "Generic (PLEG): container finished" podID="674fa73b-55ac-4158-a819-83483a19fb32" containerID="dcc1f23f89e371102fa418c12e893d25035b922c7e3ff1432674c46f35a6ddca" exitCode=0 Oct 30 21:39:44.587039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.587021 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" event={"ID":"674fa73b-55ac-4158-a819-83483a19fb32","Type":"ContainerDied","Data":"dcc1f23f89e371102fa418c12e893d25035b922c7e3ff1432674c46f35a6ddca"} Oct 30 21:39:44.589568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.589544 4526 generic.go:356] "Generic (PLEG): container finished" podID="4215fa7d-c8cc-47f3-a8f8-b7d80b64418d" containerID="1f65ac5708470d9890d9c94caa0959b18199835b3e8caa21204dc6bbfcc00ca1" exitCode=0 Oct 30 21:39:44.589744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:44.589728 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" event={"ID":"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d","Type":"ContainerDied","Data":"1f65ac5708470d9890d9c94caa0959b18199835b3e8caa21204dc6bbfcc00ca1"} Oct 30 21:39:45.060127 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:45.060082 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31bb8bee_a24f_479b_8331_bca0f146a40d.slice/crio-38bbd302b52a9dccde813a86ff9951fdfae716d65fb7eb68a5457927284da3b2 WatchSource:0}: Error finding container 38bbd302b52a9dccde813a86ff9951fdfae716d65fb7eb68a5457927284da3b2: Status 404 returned error can't find the container with id 38bbd302b52a9dccde813a86ff9951fdfae716d65fb7eb68a5457927284da3b2 Oct 30 21:39:45.064334 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.063535 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh"] Oct 30 21:39:45.166089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.165992 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:39:45.166089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.166044 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:39:45.240516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.238707 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:45.284330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.284298 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-openshift-ca\") pod \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " Oct 30 21:39:45.284472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.284403 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-kube-api-access-p24m4\" (UniqueName: \"kubernetes.io/projected/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-kube-api-access-p24m4\") pod \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " Oct 30 21:39:45.284573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.284531 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-xks-ca\") pod \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\" (UID: \"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d\") " Oct 30 21:39:45.284957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.284932 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-openshift-ca" pod "4215fa7d-c8cc-47f3-a8f8-b7d80b64418d" (UID: "4215fa7d-c8cc-47f3-a8f8-b7d80b64418d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:45.285376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.285353 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:45.286726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.286695 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-xks-ca" pod "4215fa7d-c8cc-47f3-a8f8-b7d80b64418d" (UID: "4215fa7d-c8cc-47f3-a8f8-b7d80b64418d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:45.286835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.286776 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-kube-api-access-p24m4" pod "4215fa7d-c8cc-47f3-a8f8-b7d80b64418d" (UID: "4215fa7d-c8cc-47f3-a8f8-b7d80b64418d"). InnerVolumeSpecName "kube-api-access-p24m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:45.387061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.387021 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24m4\" (UniqueName: \"kubernetes.io/projected/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-kube-api-access-p24m4\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:45.387061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.387057 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:45.468431 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.468398 4526 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:39:45.468601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.468451 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:39:45.629252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.629213 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" event={"ID":"31bb8bee-a24f-479b-8331-bca0f146a40d","Type":"ContainerStarted","Data":"38bbd302b52a9dccde813a86ff9951fdfae716d65fb7eb68a5457927284da3b2"} Oct 30 21:39:45.631729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.631697 4526 generic.go:356] "Generic (PLEG): container finished" podID="0a3ee4a0-d451-4b8b-8586-da08c01d8321" containerID="5d5c80ef69d69e8b295c7ea4e8fc2f42266c02c6d99251459df5aacbde67fda3" exitCode=0 Oct 30 21:39:45.631859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.631806 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" event={"ID":"0a3ee4a0-d451-4b8b-8586-da08c01d8321","Type":"ContainerDied","Data":"5d5c80ef69d69e8b295c7ea4e8fc2f42266c02c6d99251459df5aacbde67fda3"} Oct 30 21:39:45.633892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.633860 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" Oct 30 21:39:45.634018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.633898 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw" event={"ID":"4215fa7d-c8cc-47f3-a8f8-b7d80b64418d","Type":"ContainerDied","Data":"85f3b4f485eca2efa8939dd31aa4b3de23cf6f3a2006efc505da5d1e4a2dc0aa"} Oct 30 21:39:45.634018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.633953 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f3b4f485eca2efa8939dd31aa4b3de23cf6f3a2006efc505da5d1e4a2dc0aa" Oct 30 21:39:45.854393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.854358 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:45.930059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.909058 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-openshift-ca\") pod \"674fa73b-55ac-4158-a819-83483a19fb32\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " Oct 30 21:39:45.930059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.909227 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674fa73b-55ac-4158-a819-83483a19fb32-kube-api-access-ztgjr\" (UniqueName: \"kubernetes.io/projected/674fa73b-55ac-4158-a819-83483a19fb32-kube-api-access-ztgjr\") pod \"674fa73b-55ac-4158-a819-83483a19fb32\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " Oct 30 21:39:45.930059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.909401 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-xks-ca\") pod \"674fa73b-55ac-4158-a819-83483a19fb32\" (UID: \"674fa73b-55ac-4158-a819-83483a19fb32\") " Oct 30 21:39:45.930059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.920049 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-openshift-ca" pod "674fa73b-55ac-4158-a819-83483a19fb32" (UID: "674fa73b-55ac-4158-a819-83483a19fb32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:45.930752 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.930713 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674fa73b-55ac-4158-a819-83483a19fb32-kube-api-access-ztgjr" pod "674fa73b-55ac-4158-a819-83483a19fb32" (UID: "674fa73b-55ac-4158-a819-83483a19fb32"). InnerVolumeSpecName "kube-api-access-ztgjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:45.932941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:45.932564 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-xks-ca" pod "674fa73b-55ac-4158-a819-83483a19fb32" (UID: "674fa73b-55ac-4158-a819-83483a19fb32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:46.029627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.029530 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:46.029854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.029836 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztgjr\" (UniqueName: \"kubernetes.io/projected/674fa73b-55ac-4158-a819-83483a19fb32-kube-api-access-ztgjr\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:46.029973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.029958 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/674fa73b-55ac-4158-a819-83483a19fb32-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:46.165611 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.165566 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:39:46.165773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.165621 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 21:39:46.468270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.468233 4526 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:39:46.471405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.470365 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:39:46.648069 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.647977 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" Oct 30 21:39:46.648291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.648263 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt" event={"ID":"674fa73b-55ac-4158-a819-83483a19fb32","Type":"ContainerDied","Data":"c9d11e41025d0cdf59f7e2a5f0e70f9313bb9c0dac739bd75fdceecae10191ca"} Oct 30 21:39:46.648443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.648425 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9d11e41025d0cdf59f7e2a5f0e70f9313bb9c0dac739bd75fdceecae10191ca" Oct 30 21:39:46.658850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.658787 4526 generic.go:356] "Generic (PLEG): container finished" podID="31bb8bee-a24f-479b-8331-bca0f146a40d" containerID="bdeef5e74091189a6f47f62456166dbc145e2d912d4b772adfc7f757afdef7c6" exitCode=0 Oct 30 21:39:46.658996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.658888 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" event={"ID":"31bb8bee-a24f-479b-8331-bca0f146a40d","Type":"ContainerDied","Data":"bdeef5e74091189a6f47f62456166dbc145e2d912d4b772adfc7f757afdef7c6"} Oct 30 21:39:46.660349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.660323 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz"] Oct 30 21:39:46.661322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.661298 4526 cpu_manager.go:414] "containerMap: removing container" podUID="4215fa7d-c8cc-47f3-a8f8-b7d80b64418d" containerName="revoke-keys" Oct 30 21:39:46.661322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.661326 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4215fa7d-c8cc-47f3-a8f8-b7d80b64418d" containerName="revoke-keys" Oct 30 21:39:46.661499 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.661409 4526 cpu_manager.go:414] "containerMap: removing container" podUID="674fa73b-55ac-4158-a819-83483a19fb32" containerName="revoke-keys" Oct 30 21:39:46.661499 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.661418 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674fa73b-55ac-4158-a819-83483a19fb32" containerName="revoke-keys" Oct 30 21:39:46.661629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.661610 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="4215fa7d-c8cc-47f3-a8f8-b7d80b64418d" containerName="revoke-keys" Oct 30 21:39:46.661690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.661630 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="674fa73b-55ac-4158-a819-83483a19fb32" containerName="revoke-keys" Oct 30 21:39:46.665993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.665971 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.728637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.728593 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz"] Oct 30 21:39:46.741879 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.741839 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwvj\" (UniqueName: \"kubernetes.io/projected/7d2b5c52-ab4a-41cd-a050-320135ef3271-kube-api-access-lpwvj\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.742085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.741928 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.742172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.742152 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.843701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.843667 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.844165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.844145 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwvj\" (UniqueName: \"kubernetes.io/projected/7d2b5c52-ab4a-41cd-a050-320135ef3271-kube-api-access-lpwvj\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.844288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.844275 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.844908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.844510 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.859745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.850922 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.929958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.929857 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:46.930956 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.930929 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwvj\" (UniqueName: \"kubernetes.io/projected/7d2b5c52-ab4a-41cd-a050-320135ef3271-kube-api-access-lpwvj\") pod \"maas-api-revoke-sub-41819b1d-dnvjz\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:46.991157 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:46.991118 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:47.046555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.045991 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-xks-ca\") pod \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " Oct 30 21:39:47.046555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.046119 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-openshift-ca\") pod \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " Oct 30 21:39:47.046555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.046357 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3ee4a0-d451-4b8b-8586-da08c01d8321-kube-api-access-j6qg8\" (UniqueName: \"kubernetes.io/projected/0a3ee4a0-d451-4b8b-8586-da08c01d8321-kube-api-access-j6qg8\") pod \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\" (UID: \"0a3ee4a0-d451-4b8b-8586-da08c01d8321\") " Oct 30 21:39:47.046555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.046455 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-openshift-ca" pod "0a3ee4a0-d451-4b8b-8586-da08c01d8321" (UID: "0a3ee4a0-d451-4b8b-8586-da08c01d8321"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:47.047032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.047008 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:47.049287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.049253 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-xks-ca" pod "0a3ee4a0-d451-4b8b-8586-da08c01d8321" (UID: "0a3ee4a0-d451-4b8b-8586-da08c01d8321"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:47.049406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.049303 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3ee4a0-d451-4b8b-8586-da08c01d8321-kube-api-access-j6qg8" pod "0a3ee4a0-d451-4b8b-8586-da08c01d8321" (UID: "0a3ee4a0-d451-4b8b-8586-da08c01d8321"). InnerVolumeSpecName "kube-api-access-j6qg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:47.149529 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.149474 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3ee4a0-d451-4b8b-8586-da08c01d8321-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:47.149763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.149743 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qg8\" (UniqueName: \"kubernetes.io/projected/0a3ee4a0-d451-4b8b-8586-da08c01d8321-kube-api-access-j6qg8\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:47.402661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.402631 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz"] Oct 30 21:39:47.472405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.472366 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:47.472952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.472784 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:47.473730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.473708 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:39:47.579378 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.579342 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:47.580251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.580192 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:47.581188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.581174 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:39:47.668464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.668368 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" event={"ID":"7d2b5c52-ab4a-41cd-a050-320135ef3271","Type":"ContainerStarted","Data":"41370b45e6af6fee0c51eb3066963eb6090d9451e97b895f3e9d21bdac02a1c0"} Oct 30 21:39:47.669924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.669905 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" Oct 30 21:39:47.670044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.669954 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw" event={"ID":"0a3ee4a0-d451-4b8b-8586-da08c01d8321","Type":"ContainerDied","Data":"a33794bd67df4b6569d5ac6bf871cc82e0b2cf536e044b8fdf590202678e02f4"} Oct 30 21:39:47.670044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.669997 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a33794bd67df4b6569d5ac6bf871cc82e0b2cf536e044b8fdf590202678e02f4" Oct 30 21:39:47.857354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.857323 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:47.962499 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.962387 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-openshift-ca\") pod \"31bb8bee-a24f-479b-8331-bca0f146a40d\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " Oct 30 21:39:47.962499 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.962444 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bb8bee-a24f-479b-8331-bca0f146a40d-kube-api-access-h44vp\" (UniqueName: \"kubernetes.io/projected/31bb8bee-a24f-479b-8331-bca0f146a40d-kube-api-access-h44vp\") pod \"31bb8bee-a24f-479b-8331-bca0f146a40d\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " Oct 30 21:39:47.962724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.962546 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-xks-ca\") pod \"31bb8bee-a24f-479b-8331-bca0f146a40d\" (UID: \"31bb8bee-a24f-479b-8331-bca0f146a40d\") " Oct 30 21:39:47.962882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.962860 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-openshift-ca" pod "31bb8bee-a24f-479b-8331-bca0f146a40d" (UID: "31bb8bee-a24f-479b-8331-bca0f146a40d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:47.963167 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.963149 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:47.964938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.964916 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bb8bee-a24f-479b-8331-bca0f146a40d-kube-api-access-h44vp" pod "31bb8bee-a24f-479b-8331-bca0f146a40d" (UID: "31bb8bee-a24f-479b-8331-bca0f146a40d"). InnerVolumeSpecName "kube-api-access-h44vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:47.965137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:47.965111 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-xks-ca" pod "31bb8bee-a24f-479b-8331-bca0f146a40d" (UID: "31bb8bee-a24f-479b-8331-bca0f146a40d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:48.064136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.064104 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h44vp\" (UniqueName: \"kubernetes.io/projected/31bb8bee-a24f-479b-8331-bca0f146a40d-kube-api-access-h44vp\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:48.064136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.064137 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31bb8bee-a24f-479b-8331-bca0f146a40d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:48.159109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.159071 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz"] Oct 30 21:39:48.159842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.159825 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0a3ee4a0-d451-4b8b-8586-da08c01d8321" containerName="revoke-keys" Oct 30 21:39:48.159901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.159844 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3ee4a0-d451-4b8b-8586-da08c01d8321" containerName="revoke-keys" Oct 30 21:39:48.159901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.159861 4526 cpu_manager.go:414] "containerMap: removing container" podUID="31bb8bee-a24f-479b-8331-bca0f146a40d" containerName="revoke-keys" Oct 30 21:39:48.159901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.159866 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bb8bee-a24f-479b-8331-bca0f146a40d" containerName="revoke-keys" Oct 30 21:39:48.160001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.159964 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3ee4a0-d451-4b8b-8586-da08c01d8321" containerName="revoke-keys" Oct 30 21:39:48.160001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.159974 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bb8bee-a24f-479b-8331-bca0f146a40d" containerName="revoke-keys" Oct 30 21:39:48.163803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.163785 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.217021 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.216678 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-89c0fd-openshift-default-dockercfg-kzh9g\"" Oct 30 21:39:48.222053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.222027 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz"] Oct 30 21:39:48.266384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266355 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.266538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266406 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.266538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266517 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cxq4\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-kube-api-access-5cxq4\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.266658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266636 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c9219-9447-4846-a80c-5adb4466afde-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.266691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266682 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.266752 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266740 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.266790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266759 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.266865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266852 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c9219-9447-4846-a80c-5adb4466afde-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.266914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.266903 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.368635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.368600 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.368796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.368672 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxq4\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-kube-api-access-5cxq4\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.368796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.368768 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c9219-9447-4846-a80c-5adb4466afde-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.368884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.368812 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.368884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.368854 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.368884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.368879 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.369104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.368970 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c9219-9447-4846-a80c-5adb4466afde-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.369104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.369043 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.369211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.369150 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.369262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.369215 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.369331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.369304 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.369450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.369419 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.369683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.369662 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.369818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.369802 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c9219-9447-4846-a80c-5adb4466afde-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.371177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.371158 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.371460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.371442 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c9219-9447-4846-a80c-5adb4466afde-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.425247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.425219 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.425363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.425300 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cxq4\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-kube-api-access-5cxq4\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.479481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.479400 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:48.678803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.678766 4526 generic.go:356] "Generic (PLEG): container finished" podID="7d2b5c52-ab4a-41cd-a050-320135ef3271" containerID="24b0c6e7caf98f0afb6c4fcac6d7740c458916ac9617502cea1a87e9980d3971" exitCode=0 Oct 30 21:39:48.678960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.678851 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" event={"ID":"7d2b5c52-ab4a-41cd-a050-320135ef3271","Type":"ContainerDied","Data":"24b0c6e7caf98f0afb6c4fcac6d7740c458916ac9617502cea1a87e9980d3971"} Oct 30 21:39:48.680248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.680231 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" Oct 30 21:39:48.680361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.680284 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh" event={"ID":"31bb8bee-a24f-479b-8331-bca0f146a40d","Type":"ContainerDied","Data":"38bbd302b52a9dccde813a86ff9951fdfae716d65fb7eb68a5457927284da3b2"} Oct 30 21:39:48.680361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.680329 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38bbd302b52a9dccde813a86ff9951fdfae716d65fb7eb68a5457927284da3b2" Oct 30 21:39:48.965913 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.965835 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz"] Oct 30 21:39:48.967239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.967204 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:48.967337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.967283 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:48.967337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:48.967327 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:49.722613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:49.722574 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" event={"ID":"8e5c9219-9447-4846-a80c-5adb4466afde","Type":"ContainerStarted","Data":"69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa"} Oct 30 21:39:49.723292 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:49.723261 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" event={"ID":"8e5c9219-9447-4846-a80c-5adb4466afde","Type":"ContainerStarted","Data":"e13f1c379ce52487ec829237e889ada3fbc4653f80df582e069ec86aa7ba9a35"} Oct 30 21:39:49.842648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:49.842594 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podStartSLOduration=1.842578131 podStartE2EDuration="1.842578131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:49.839243546 +0000 UTC m=+1379.972616762" watchObservedRunningTime="2026-10-30 21:39:49.842578131 +0000 UTC m=+1379.975951329" Oct 30 21:39:49.890718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:49.890694 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:49.994641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:49.994527 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2b5c52-ab4a-41cd-a050-320135ef3271-kube-api-access-lpwvj\" (UniqueName: \"kubernetes.io/projected/7d2b5c52-ab4a-41cd-a050-320135ef3271-kube-api-access-lpwvj\") pod \"7d2b5c52-ab4a-41cd-a050-320135ef3271\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " Oct 30 21:39:49.994934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:49.994916 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-openshift-ca\") pod \"7d2b5c52-ab4a-41cd-a050-320135ef3271\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " Oct 30 21:39:49.995081 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:49.995067 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-xks-ca\") pod \"7d2b5c52-ab4a-41cd-a050-320135ef3271\" (UID: \"7d2b5c52-ab4a-41cd-a050-320135ef3271\") " Oct 30 21:39:49.997232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:49.997204 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-openshift-ca" pod "7d2b5c52-ab4a-41cd-a050-320135ef3271" (UID: "7d2b5c52-ab4a-41cd-a050-320135ef3271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:50.036539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.032120 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2b5c52-ab4a-41cd-a050-320135ef3271-kube-api-access-lpwvj" pod "7d2b5c52-ab4a-41cd-a050-320135ef3271" (UID: "7d2b5c52-ab4a-41cd-a050-320135ef3271"). InnerVolumeSpecName "kube-api-access-lpwvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:50.036539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.034238 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-xks-ca" pod "7d2b5c52-ab4a-41cd-a050-320135ef3271" (UID: "7d2b5c52-ab4a-41cd-a050-320135ef3271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:50.098723 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.098684 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpwvj\" (UniqueName: \"kubernetes.io/projected/7d2b5c52-ab4a-41cd-a050-320135ef3271-kube-api-access-lpwvj\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:50.099746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.099720 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:50.099933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.099918 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2b5c52-ab4a-41cd-a050-320135ef3271-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:50.250276 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.250188 4526 scope.go:122] "RemoveContainer" containerID="4b34186f7a6e8a3a52d23650495c69455bfb30573a3b24dff4d3deaf02476adc" Oct 30 21:39:50.261385 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.261363 4526 scope.go:122] "RemoveContainer" containerID="ea4401ac789f3408cc253dce6ca2898ff25822cc2f300212d41dddf29f7a029e" Oct 30 21:39:50.271013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.270990 4526 scope.go:122] "RemoveContainer" containerID="6feb599eecb45174152529c401df3f09cb38ba18b4caba9397b80fe7500787c8" Oct 30 21:39:50.280334 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.280311 4526 scope.go:122] "RemoveContainer" containerID="8e1e8a09eab3b1e87d5879f8104335b11289ebf85feecc39302ee6d17181c862" Oct 30 21:39:50.289567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.289548 4526 scope.go:122] "RemoveContainer" containerID="20435decf13ffa938bdec57a71b4ac7f85ded45168fefc570f14e39e23cf8f18" Oct 30 21:39:50.326438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.326415 4526 scope.go:122] "RemoveContainer" containerID="1122d9772d02abf3cd9251d2146add48d1e2af62c9185ab8f1e76b86386618c5" Oct 30 21:39:50.336668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.336646 4526 scope.go:122] "RemoveContainer" containerID="0f53de66fd17c8b508896c754705d57dcf45db5dc905fcc8d9223380595eede9" Oct 30 21:39:50.346046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.346020 4526 scope.go:122] "RemoveContainer" containerID="e4591bfd7dc7967a6aa27427cccf083645a5e8255d68491c9dd63151e662c2f3" Oct 30 21:39:50.355361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.355343 4526 scope.go:122] "RemoveContainer" containerID="5a9e03f54ccee88b21e4f064331638ca64f83a66fa2aaf0ea9e8ea3ab476cf2d" Oct 30 21:39:50.367183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.367164 4526 scope.go:122] "RemoveContainer" containerID="473f9d5b7188a77d1b8126f404a59f3293e7aeff6b1d906ef9f7247bff824de1" Oct 30 21:39:50.480161 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.480137 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:39:50.481997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.481975 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:50.482062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.482021 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:50.732241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.732215 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" Oct 30 21:39:50.732710 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.732245 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz" event={"ID":"7d2b5c52-ab4a-41cd-a050-320135ef3271","Type":"ContainerDied","Data":"41370b45e6af6fee0c51eb3066963eb6090d9451e97b895f3e9d21bdac02a1c0"} Oct 30 21:39:50.732710 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:50.732294 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41370b45e6af6fee0c51eb3066963eb6090d9451e97b895f3e9d21bdac02a1c0" Oct 30 21:39:51.480153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:51.480122 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:51.480306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:51.480180 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:52.435771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.435724 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn"] Oct 30 21:39:52.436460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.436444 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7d2b5c52-ab4a-41cd-a050-320135ef3271" containerName="revoke-keys" Oct 30 21:39:52.436544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.436461 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2b5c52-ab4a-41cd-a050-320135ef3271" containerName="revoke-keys" Oct 30 21:39:52.436672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.436658 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2b5c52-ab4a-41cd-a050-320135ef3271" containerName="revoke-keys" Oct 30 21:39:52.442326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.442310 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.479834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.479800 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:52.479987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.479855 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:52.523704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.523665 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn"] Oct 30 21:39:52.540821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.540784 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:52.541014 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.540977 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:39:52.541115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.541060 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:39:52.541211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.541129 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:39:52.546750 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.546726 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.546880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.546863 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbm5g\" (UniqueName: \"kubernetes.io/projected/21a1e85d-2b35-4222-bcfa-80d27a294234-kube-api-access-qbm5g\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.547011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.546998 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.647894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.647864 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.648068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.647966 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbm5g\" (UniqueName: \"kubernetes.io/projected/21a1e85d-2b35-4222-bcfa-80d27a294234-kube-api-access-qbm5g\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.648110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.648072 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.648741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.648691 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.650320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.650300 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.719740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.713159 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbm5g\" (UniqueName: \"kubernetes.io/projected/21a1e85d-2b35-4222-bcfa-80d27a294234-kube-api-access-qbm5g\") pod \"maas-api-revoke-sub-663fb99c-57dsn\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:52.756251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:52.756225 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:53.164873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:53.164847 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn"] Oct 30 21:39:53.165353 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:39:53.165334 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a1e85d_2b35_4222_bcfa_80d27a294234.slice/crio-74c66ff62b212047b1f408d2ddd237b1554c5518c74e373681c7322a8d542b2e WatchSource:0}: Error finding container 74c66ff62b212047b1f408d2ddd237b1554c5518c74e373681c7322a8d542b2e: Status 404 returned error can't find the container with id 74c66ff62b212047b1f408d2ddd237b1554c5518c74e373681c7322a8d542b2e Oct 30 21:39:53.480362 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:53.480322 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:53.480789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:53.480388 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:53.781423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:53.781336 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" event={"ID":"21a1e85d-2b35-4222-bcfa-80d27a294234","Type":"ContainerStarted","Data":"74c66ff62b212047b1f408d2ddd237b1554c5518c74e373681c7322a8d542b2e"} Oct 30 21:39:54.491785 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:54.491748 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:54.492366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:54.492331 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:54.825515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:54.821945 4526 generic.go:356] "Generic (PLEG): container finished" podID="21a1e85d-2b35-4222-bcfa-80d27a294234" containerID="2efc8b92cc8c6c9bb2dd5869f7b4a6b974fad9b252218ca67eee1533e0073939" exitCode=0 Oct 30 21:39:54.825515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:54.822008 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" event={"ID":"21a1e85d-2b35-4222-bcfa-80d27a294234","Type":"ContainerDied","Data":"2efc8b92cc8c6c9bb2dd5869f7b4a6b974fad9b252218ca67eee1533e0073939"} Oct 30 21:39:54.950637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:54.950586 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:39:54.979130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:54.979084 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:39:54.984686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:54.979388 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.029633 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.029582 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.204577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.188748 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.204577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.200091 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 21:39:55.226648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.226597 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.356734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.356676 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.481170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.481061 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:55.481442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.481397 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:55.484632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.484591 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.636675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.636615 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.651739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.651646 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.909704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.909645 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.956822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.956758 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.958439 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.957224 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:39:55.996513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.992628 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.996513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.993047 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:39:55.996513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.994439 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:39:55.996513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.994803 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:39:55.996513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:55.994861 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:39:56.034807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.034759 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.094529 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.083921 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:39:56.303538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.294690 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.303538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.295381 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:39:56.354515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.311418 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 30 21:39:56.463806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.446441 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:39:56.499006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.498955 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.557583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.532881 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:39:56.728711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.728663 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.744767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.744723 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.781540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.745466 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.860979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:56.860891 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:39:57.021518 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.012627 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 21:39:57.104521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.101698 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:57.104521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.101772 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.104521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.102552 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:39:57.326517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.308051 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:57.326517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.308280 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:39:57.326517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.309841 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": context deadline exceeded" Oct 30 21:39:57.326517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.310700 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:39:57.348515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.327181 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:39:57.348515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.343210 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:57.348515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.343267 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:57.422566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.420421 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.477573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.471576 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.548515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.501506 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z" podUID="3f6bfaff-39af-4c61-a3b5-5d25d0098a86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded" Oct 30 21:39:57.548515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.517791 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:39:57.575736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.554630 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:39:57.575736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.555794 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:57.575736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.555846 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:57.575736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.563236 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.695515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.679581 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.769365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.648504 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:39:57.782868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.779793 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:39:57.782868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.781991 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr" podUID="c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": context deadline exceeded" Oct 30 21:39:57.782868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.782205 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.829519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.826002 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:57.884590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.883570 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.938514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.937613 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-xks-ca\") pod \"21a1e85d-2b35-4222-bcfa-80d27a294234\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " Oct 30 21:39:57.938514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.937661 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a1e85d-2b35-4222-bcfa-80d27a294234-kube-api-access-qbm5g\" (UniqueName: \"kubernetes.io/projected/21a1e85d-2b35-4222-bcfa-80d27a294234-kube-api-access-qbm5g\") pod \"21a1e85d-2b35-4222-bcfa-80d27a294234\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " Oct 30 21:39:57.938514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.937850 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-openshift-ca\") pod \"21a1e85d-2b35-4222-bcfa-80d27a294234\" (UID: \"21a1e85d-2b35-4222-bcfa-80d27a294234\") " Oct 30 21:39:57.947771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.947730 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-openshift-ca" pod "21a1e85d-2b35-4222-bcfa-80d27a294234" (UID: "21a1e85d-2b35-4222-bcfa-80d27a294234"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:57.953219 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:57.949962 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.033007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.032947 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 21:39:58.221641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.218682 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-xks-ca" pod "21a1e85d-2b35-4222-bcfa-80d27a294234" (UID: "21a1e85d-2b35-4222-bcfa-80d27a294234"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:58.325920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.325748 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21a1e85d-2b35-4222-bcfa-80d27a294234-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.329663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.329619 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.430514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.424176 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a1e85d-2b35-4222-bcfa-80d27a294234-kube-api-access-qbm5g" pod "21a1e85d-2b35-4222-bcfa-80d27a294234" (UID: "21a1e85d-2b35-4222-bcfa-80d27a294234"). InnerVolumeSpecName "kube-api-access-qbm5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:58.430514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.424421 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.461453 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.461416 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbm5g\" (UniqueName: \"kubernetes.io/projected/21a1e85d-2b35-4222-bcfa-80d27a294234-kube-api-access-qbm5g\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.492573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.462383 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:39:58.492573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.490631 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:58.492573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.490681 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:58.492573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.491217 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.536515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.534707 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:58.536515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.534805 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.551546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.544692 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.551546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.547338 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:58.551546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.547436 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.555654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.555611 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.584286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.571159 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.615764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.605058 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.729520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.718627 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.771723 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.771622 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.791661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.791611 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:39:58.887645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.887597 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.990516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.986600 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" event={"ID":"21a1e85d-2b35-4222-bcfa-80d27a294234","Type":"ContainerDied","Data":"74c66ff62b212047b1f408d2ddd237b1554c5518c74e373681c7322a8d542b2e"} Oct 30 21:39:58.990516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.986680 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c66ff62b212047b1f408d2ddd237b1554c5518c74e373681c7322a8d542b2e" Oct 30 21:39:58.990516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:58.986764 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn" Oct 30 21:39:59.042041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.041997 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.328575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.254682 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" podUID="cdf20d6e-28cb-40ef-ba2b-b2108efbe6da" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.455566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.454742 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.498945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.498914 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 21:39:59.499227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.499194 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 21:39:59.501405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.501378 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 30 21:39:59.501866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.501843 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:39:59.557716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.557672 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:59.563118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.563082 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.573986 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.573625 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.609540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.608576 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.732515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.732075 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.774744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.774697 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.808516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.792728 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.915858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:39:59.915757 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.055516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.042664 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.475514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.469396 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.603515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.524278 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:00.603515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.524335 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.603515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.600855 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.665514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.651144 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:40:00.666902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.666853 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 30 21:40:00.744709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.744590 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:40:00.745550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.743924 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.746058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.743998 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.770514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.750630 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:40:00.793069 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.793008 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Oct 30 21:40:00.852427 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.795660 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:40:00.852737 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.846514 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.961043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.961005 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:40:00.962513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.961571 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:40:00.962513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.961631 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:40:00.962513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.961677 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:40:00.966980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.965674 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:40:00.971769 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.971741 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:40:00.974719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.974696 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:40:00.975100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.975078 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:40:00.977840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.977816 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:40:00.990033 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:00.990017 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:40:01.002690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:01.002610 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:40:01.008853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:01.008829 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:40:01.186878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:01.186791 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:40:01.187272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:01.187250 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:40:01.188342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:01.188319 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:40:06.759064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.759027 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn"] Oct 30 21:40:06.759792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.759773 4526 cpu_manager.go:414] "containerMap: removing container" podUID="21a1e85d-2b35-4222-bcfa-80d27a294234" containerName="revoke-keys" Oct 30 21:40:06.759792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.759792 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a1e85d-2b35-4222-bcfa-80d27a294234" containerName="revoke-keys" Oct 30 21:40:06.759971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.759923 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a1e85d-2b35-4222-bcfa-80d27a294234" containerName="revoke-keys" Oct 30 21:40:06.765321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.765292 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.795416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.795386 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.795609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.795433 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9l5w\" (UniqueName: \"kubernetes.io/projected/378c3fe9-9948-4a53-9729-92cdd552b886-kube-api-access-q9l5w\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.795609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.795553 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.820272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.820237 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn"] Oct 30 21:40:06.862770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.862742 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:06.862932 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.862812 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:06.862932 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.862807 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:06.866588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.866575 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:40:06.896946 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.896915 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.897138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.896970 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9l5w\" (UniqueName: \"kubernetes.io/projected/378c3fe9-9948-4a53-9729-92cdd552b886-kube-api-access-q9l5w\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.897138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.897051 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.897592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.897561 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.899501 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.899464 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:06.953290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:06.953260 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9l5w\" (UniqueName: \"kubernetes.io/projected/378c3fe9-9948-4a53-9729-92cdd552b886-kube-api-access-q9l5w\") pod \"maas-api-revoke-sub-db12a902-lzbcn\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:07.084210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:07.084181 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:07.467279 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:07.467247 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378c3fe9_9948_4a53_9729_92cdd552b886.slice/crio-dee6e416ece448b76b86380ccac48e8b07b7ae340ccee46f272446f9d9f4a739 WatchSource:0}: Error finding container dee6e416ece448b76b86380ccac48e8b07b7ae340ccee46f272446f9d9f4a739: Status 404 returned error can't find the container with id dee6e416ece448b76b86380ccac48e8b07b7ae340ccee46f272446f9d9f4a739 Oct 30 21:40:07.467840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:07.467819 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn"] Oct 30 21:40:08.179100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.179056 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" event={"ID":"378c3fe9-9948-4a53-9729-92cdd552b886","Type":"ContainerStarted","Data":"298bc9eca67cc95342a7696b8625fa6141db4c71f859151aeeb04fe37032776a"} Oct 30 21:40:08.179649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.179111 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" event={"ID":"378c3fe9-9948-4a53-9729-92cdd552b886","Type":"ContainerStarted","Data":"dee6e416ece448b76b86380ccac48e8b07b7ae340ccee46f272446f9d9f4a739"} Oct 30 21:40:08.285928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.285893 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql"] Oct 30 21:40:08.290559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.290531 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.304452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.304394 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" podStartSLOduration=2.30437555 podStartE2EDuration="2.30437555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:08.2933941 +0000 UTC m=+1398.426767289" watchObservedRunningTime="2026-10-30 21:40:08.30437555 +0000 UTC m=+1398.437748749" Oct 30 21:40:08.319753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.319716 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql"] Oct 30 21:40:08.420404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.420372 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.420557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.420418 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcztq\" (UniqueName: \"kubernetes.io/projected/b84471d9-a81d-48f3-aed7-91bdd7bca242-kube-api-access-bcztq\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.420612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.420600 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.522076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.521977 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.522076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.522021 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcztq\" (UniqueName: \"kubernetes.io/projected/b84471d9-a81d-48f3-aed7-91bdd7bca242-kube-api-access-bcztq\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.522393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.522134 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.522841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.522817 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.524681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.524648 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.581752 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.581720 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcztq\" (UniqueName: \"kubernetes.io/projected/b84471d9-a81d-48f3-aed7-91bdd7bca242-kube-api-access-bcztq\") pod \"maas-api-revoke-sub-e180a309-4bhql\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:08.608844 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:08.608808 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:09.086995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.086954 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9"] Oct 30 21:40:09.092430 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.092405 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.183582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.183459 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9"] Oct 30 21:40:09.194704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.194670 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.198546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.198469 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.198866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.198824 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98tk9\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-kube-api-access-98tk9\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.199085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.199052 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.199292 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.199275 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0b7ee-7835-4c92-829e-f734300c96b5-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.199530 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.199513 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.199713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.199697 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.199945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.199928 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.206673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.206629 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.227583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.227549 4526 generic.go:356] "Generic (PLEG): container finished" podID="378c3fe9-9948-4a53-9729-92cdd552b886" containerID="298bc9eca67cc95342a7696b8625fa6141db4c71f859151aeeb04fe37032776a" exitCode=0 Oct 30 21:40:09.228302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.227835 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" event={"ID":"378c3fe9-9948-4a53-9729-92cdd552b886","Type":"ContainerDied","Data":"298bc9eca67cc95342a7696b8625fa6141db4c71f859151aeeb04fe37032776a"} Oct 30 21:40:09.245684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.240795 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql"] Oct 30 21:40:09.261986 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:09.261865 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84471d9_a81d_48f3_aed7_91bdd7bca242.slice/crio-c814f4193e8cc838079f431e8a9858a01a6179a1d60b7f5005a045706250fe72 WatchSource:0}: Error finding container c814f4193e8cc838079f431e8a9858a01a6179a1d60b7f5005a045706250fe72: Status 404 returned error can't find the container with id c814f4193e8cc838079f431e8a9858a01a6179a1d60b7f5005a045706250fe72 Oct 30 21:40:09.308438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.308403 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.308705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.308657 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.308825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.308752 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.308825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.308788 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98tk9\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-kube-api-access-98tk9\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.308937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.308834 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.308937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.308857 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0b7ee-7835-4c92-829e-f734300c96b5-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.308937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.308916 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.309074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.308951 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.309074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.309023 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.309520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.309452 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.310635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.310071 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0b7ee-7835-4c92-829e-f734300c96b5-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.310635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.310592 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.310795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.310681 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.310851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.310818 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.313938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.313914 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.315744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.315717 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.370912 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.370831 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.371062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.371017 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98tk9\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-kube-api-access-98tk9\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.408118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.408085 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:09.934594 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.933406 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:09.934594 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.933591 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:09.934594 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.933690 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:09.940715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:09.936994 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9"] Oct 30 21:40:10.240431 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.240330 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" event={"ID":"75f0b7ee-7835-4c92-829e-f734300c96b5","Type":"ContainerStarted","Data":"5faca56c00f6f1063238fea6620a22ca2ba12d9c0705a9deeed4573e7e3c4643"} Oct 30 21:40:10.240431 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.240398 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" event={"ID":"75f0b7ee-7835-4c92-829e-f734300c96b5","Type":"ContainerStarted","Data":"58ff6df9dd7e0e4696b410a45f123cf78ec4275a89cc07cdb542fc2f652edc00"} Oct 30 21:40:10.242168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.242138 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" event={"ID":"b84471d9-a81d-48f3-aed7-91bdd7bca242","Type":"ContainerStarted","Data":"aa3376ec79b5f47cdb73a2a92a5d0eca839962b9dfe1004f9e6342f487801542"} Oct 30 21:40:10.242286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.242193 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" event={"ID":"b84471d9-a81d-48f3-aed7-91bdd7bca242","Type":"ContainerStarted","Data":"c814f4193e8cc838079f431e8a9858a01a6179a1d60b7f5005a045706250fe72"} Oct 30 21:40:10.363346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.363289 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" podStartSLOduration=1.363267551 podStartE2EDuration="1.363267551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:10.356729079 +0000 UTC m=+1400.490102278" watchObservedRunningTime="2026-10-30 21:40:10.363267551 +0000 UTC m=+1400.496640751" Oct 30 21:40:10.407707 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.407679 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:10.408763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.408745 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:10.434732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.434693 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-xks-ca\") pod \"378c3fe9-9948-4a53-9729-92cdd552b886\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " Oct 30 21:40:10.434925 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.434878 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378c3fe9-9948-4a53-9729-92cdd552b886-kube-api-access-q9l5w\" (UniqueName: \"kubernetes.io/projected/378c3fe9-9948-4a53-9729-92cdd552b886-kube-api-access-q9l5w\") pod \"378c3fe9-9948-4a53-9729-92cdd552b886\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " Oct 30 21:40:10.435000 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.434958 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-openshift-ca\") pod \"378c3fe9-9948-4a53-9729-92cdd552b886\" (UID: \"378c3fe9-9948-4a53-9729-92cdd552b886\") " Oct 30 21:40:10.435380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.435348 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-openshift-ca" pod "378c3fe9-9948-4a53-9729-92cdd552b886" (UID: "378c3fe9-9948-4a53-9729-92cdd552b886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:10.435747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.435726 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.437105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.437083 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-xks-ca" pod "378c3fe9-9948-4a53-9729-92cdd552b886" (UID: "378c3fe9-9948-4a53-9729-92cdd552b886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:10.437329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.437311 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378c3fe9-9948-4a53-9729-92cdd552b886-kube-api-access-q9l5w" pod "378c3fe9-9948-4a53-9729-92cdd552b886" (UID: "378c3fe9-9948-4a53-9729-92cdd552b886"). InnerVolumeSpecName "kube-api-access-q9l5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:10.469535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.469441 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" podStartSLOduration=2.469425476 podStartE2EDuration="2.469425476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:10.46874929 +0000 UTC m=+1400.602122469" watchObservedRunningTime="2026-10-30 21:40:10.469425476 +0000 UTC m=+1400.602798715" Oct 30 21:40:10.537038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.537009 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9l5w\" (UniqueName: \"kubernetes.io/projected/378c3fe9-9948-4a53-9729-92cdd552b886-kube-api-access-q9l5w\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:10.537190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:10.537042 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/378c3fe9-9948-4a53-9729-92cdd552b886-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:11.252604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:11.252522 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" Oct 30 21:40:11.253004 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:11.252526 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn" event={"ID":"378c3fe9-9948-4a53-9729-92cdd552b886","Type":"ContainerDied","Data":"dee6e416ece448b76b86380ccac48e8b07b7ae340ccee46f272446f9d9f4a739"} Oct 30 21:40:11.253004 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:11.252703 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee6e416ece448b76b86380ccac48e8b07b7ae340ccee46f272446f9d9f4a739" Oct 30 21:40:11.254370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:11.254342 4526 generic.go:356] "Generic (PLEG): container finished" podID="b84471d9-a81d-48f3-aed7-91bdd7bca242" containerID="aa3376ec79b5f47cdb73a2a92a5d0eca839962b9dfe1004f9e6342f487801542" exitCode=0 Oct 30 21:40:11.254542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:11.254480 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" event={"ID":"b84471d9-a81d-48f3-aed7-91bdd7bca242","Type":"ContainerDied","Data":"aa3376ec79b5f47cdb73a2a92a5d0eca839962b9dfe1004f9e6342f487801542"} Oct 30 21:40:11.422515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:11.412592 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:40:11.422515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:11.412658 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:40:12.234918 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.234880 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n"] Oct 30 21:40:12.235568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.235529 4526 cpu_manager.go:414] "containerMap: removing container" podUID="378c3fe9-9948-4a53-9729-92cdd552b886" containerName="revoke-keys" Oct 30 21:40:12.235568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.235545 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378c3fe9-9948-4a53-9729-92cdd552b886" containerName="revoke-keys" Oct 30 21:40:12.235680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.235661 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="378c3fe9-9948-4a53-9729-92cdd552b886" containerName="revoke-keys" Oct 30 21:40:12.239570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.239533 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.262702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.262684 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838b810-8246-4b65-a450-cc33e4cc1121-istiod-ca-cert\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.262984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.262731 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-istio-token\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.262984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.262869 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-envoy\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.263109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.262997 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjf8x\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-kube-api-access-bjf8x\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.263109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.263036 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838b810-8246-4b65-a450-cc33e4cc1121-istio-podinfo\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.263343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.263322 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-certs\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.263476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.263443 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-data\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.263568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.263550 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-socket\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.263627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.263595 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-credential-socket\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.343958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.342768 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-385396af-openshift-default-dockercfg-gwt67\"" Oct 30 21:40:12.347576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.347331 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n"] Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.366896 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838b810-8246-4b65-a450-cc33e4cc1121-istiod-ca-cert\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.367040 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-istio-token\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.367091 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-envoy\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.367191 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjf8x\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-kube-api-access-bjf8x\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.367226 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838b810-8246-4b65-a450-cc33e4cc1121-istio-podinfo\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.367303 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-certs\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.367453 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-data\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.367515 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-socket\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.367573 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-credential-socket\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.368151 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-credential-socket\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.370548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.368875 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838b810-8246-4b65-a450-cc33e4cc1121-istiod-ca-cert\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.371294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.370639 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-certs\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.374513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.371670 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-data\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.374513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.371941 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-socket\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.390514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.390406 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838b810-8246-4b65-a450-cc33e4cc1121-istio-podinfo\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.412564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.412081 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-envoy\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.412564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.412524 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:40:12.412832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.412575 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:40:12.465794 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.465765 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-istio-token\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.474845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.474759 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjf8x\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-kube-api-access-bjf8x\") pod \"e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.566721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.561997 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:12.587359 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:40:12.587320 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:40:12.648356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.648090 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:12.778591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.774522 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-xks-ca\") pod \"b84471d9-a81d-48f3-aed7-91bdd7bca242\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " Oct 30 21:40:12.778591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.774639 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84471d9-a81d-48f3-aed7-91bdd7bca242-kube-api-access-bcztq\" (UniqueName: \"kubernetes.io/projected/b84471d9-a81d-48f3-aed7-91bdd7bca242-kube-api-access-bcztq\") pod \"b84471d9-a81d-48f3-aed7-91bdd7bca242\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " Oct 30 21:40:12.778591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.774889 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-openshift-ca\") pod \"b84471d9-a81d-48f3-aed7-91bdd7bca242\" (UID: \"b84471d9-a81d-48f3-aed7-91bdd7bca242\") " Oct 30 21:40:12.778591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.777004 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-openshift-ca" pod "b84471d9-a81d-48f3-aed7-91bdd7bca242" (UID: "b84471d9-a81d-48f3-aed7-91bdd7bca242"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:12.792520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.786978 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-xks-ca" pod "b84471d9-a81d-48f3-aed7-91bdd7bca242" (UID: "b84471d9-a81d-48f3-aed7-91bdd7bca242"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:12.792520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.788667 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84471d9-a81d-48f3-aed7-91bdd7bca242-kube-api-access-bcztq" pod "b84471d9-a81d-48f3-aed7-91bdd7bca242" (UID: "b84471d9-a81d-48f3-aed7-91bdd7bca242"). InnerVolumeSpecName "kube-api-access-bcztq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:12.876890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.876811 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:12.876890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.876838 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b84471d9-a81d-48f3-aed7-91bdd7bca242-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:12.876890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:12.876848 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcztq\" (UniqueName: \"kubernetes.io/projected/b84471d9-a81d-48f3-aed7-91bdd7bca242-kube-api-access-bcztq\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:13.061247 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:13.061217 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0838b810_8246_4b65_a450_cc33e4cc1121.slice/crio-eaa21b06f8110f47f57c0ffdf3f3c6a7f221b145465f578a24c828296cc85297 WatchSource:0}: Error finding container eaa21b06f8110f47f57c0ffdf3f3c6a7f221b145465f578a24c828296cc85297: Status 404 returned error can't find the container with id eaa21b06f8110f47f57c0ffdf3f3c6a7f221b145465f578a24c828296cc85297 Oct 30 21:40:13.062835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.062811 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n"] Oct 30 21:40:13.064036 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.064008 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:13.064085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.064070 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:13.064119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.064100 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:13.273555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.273469 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" event={"ID":"0838b810-8246-4b65-a450-cc33e4cc1121","Type":"ContainerStarted","Data":"e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed"} Oct 30 21:40:13.274571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.274514 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" event={"ID":"0838b810-8246-4b65-a450-cc33e4cc1121","Type":"ContainerStarted","Data":"eaa21b06f8110f47f57c0ffdf3f3c6a7f221b145465f578a24c828296cc85297"} Oct 30 21:40:13.280321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.275809 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" event={"ID":"b84471d9-a81d-48f3-aed7-91bdd7bca242","Type":"ContainerDied","Data":"c814f4193e8cc838079f431e8a9858a01a6179a1d60b7f5005a045706250fe72"} Oct 30 21:40:13.280321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.275850 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c814f4193e8cc838079f431e8a9858a01a6179a1d60b7f5005a045706250fe72" Oct 30 21:40:13.280321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.275853 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql" Oct 30 21:40:13.410459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.409930 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:40:13.410459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.410008 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:40:13.424659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.424546 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" podStartSLOduration=1.4245241229999999 podStartE2EDuration="1.424524123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:13.391559543 +0000 UTC m=+1403.524932740" watchObservedRunningTime="2026-10-30 21:40:13.424524123 +0000 UTC m=+1403.557897319" Oct 30 21:40:13.565513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:13.563639 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:14.416515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:14.409881 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:40:14.416515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:14.409972 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:40:14.569654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:14.569627 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:15.291921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:15.291892 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:15.293067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:15.293046 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:40:15.419981 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:15.419944 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:40:15.420402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:15.420007 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:40:16.408820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:16.408785 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:40:16.409007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:16.408837 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 30 21:40:17.494763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.494731 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:17.496443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.496394 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:17.498008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.497683 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:40:17.730243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.730210 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n"] Oct 30 21:40:17.730925 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.730907 4526 cpu_manager.go:414] "containerMap: removing container" podUID="b84471d9-a81d-48f3-aed7-91bdd7bca242" containerName="revoke-keys" Oct 30 21:40:17.730925 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.730926 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84471d9-a81d-48f3-aed7-91bdd7bca242" containerName="revoke-keys" Oct 30 21:40:17.731288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.731034 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84471d9-a81d-48f3-aed7-91bdd7bca242" containerName="revoke-keys" Oct 30 21:40:17.745385 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.745308 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.786819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.786788 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n"] Oct 30 21:40:17.806436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.805264 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.806908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.806876 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.807145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.807128 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwjdd\" (UniqueName: \"kubernetes.io/projected/7418a23c-fab4-4512-89d2-fad697a23192-kube-api-access-gwjdd\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.831356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.831250 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:40:17.847177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.832463 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:17.847389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.832555 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:17.847524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.833564 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:17.909300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.909263 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjdd\" (UniqueName: \"kubernetes.io/projected/7418a23c-fab4-4512-89d2-fad697a23192-kube-api-access-gwjdd\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.909536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.909403 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.911312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.909630 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.911312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.910505 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.916907 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.913632 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:17.976570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:17.974311 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwjdd\" (UniqueName: \"kubernetes.io/projected/7418a23c-fab4-4512-89d2-fad697a23192-kube-api-access-gwjdd\") pod \"maas-api-revoke-sub-3ba9f570-vxw9n\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:18.073905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.073869 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:18.100393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.100354 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl"] Oct 30 21:40:18.105831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.105806 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.169400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.169365 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl"] Oct 30 21:40:18.218629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.218594 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.218629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.218637 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7061e0-467e-4274-af65-b4485c807847-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.218851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.218812 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.218892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.218858 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7061e0-467e-4274-af65-b4485c807847-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.218975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.218961 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.219025 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.218980 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.219092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.219073 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjd6b\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-kube-api-access-qjd6b\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.219137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.219098 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.219179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.219146 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321055 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321108 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7061e0-467e-4274-af65-b4485c807847-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321218 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321243 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321337 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjd6b\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-kube-api-access-qjd6b\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321381 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321441 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321521 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321551 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7061e0-467e-4274-af65-b4485c807847-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321556 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321567 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321646 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.321964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.321904 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.322139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.322124 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7061e0-467e-4274-af65-b4485c807847-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.323889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.323865 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.324182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.324138 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7061e0-467e-4274-af65-b4485c807847-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.377778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.377748 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjd6b\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-kube-api-access-qjd6b\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.377926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.377758 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.427951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.427924 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:18.457639 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:18.457601 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7418a23c_fab4_4512_89d2_fad697a23192.slice/crio-c7c888c2ffe76869646bb3ad144e5f3b1b779be53eccfcbcf2e28cbc144723b6 WatchSource:0}: Error finding container c7c888c2ffe76869646bb3ad144e5f3b1b779be53eccfcbcf2e28cbc144723b6: Status 404 returned error can't find the container with id c7c888c2ffe76869646bb3ad144e5f3b1b779be53eccfcbcf2e28cbc144723b6 Oct 30 21:40:18.460096 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.460063 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n"] Oct 30 21:40:18.815790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.815764 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl"] Oct 30 21:40:18.816185 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:18.815891 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7061e0_467e_4274_af65_b4485c807847.slice/crio-ac34e39f904f6c3223122b05c3b47a7f54086789b49cca4feadd20aa88c04c74 WatchSource:0}: Error finding container ac34e39f904f6c3223122b05c3b47a7f54086789b49cca4feadd20aa88c04c74: Status 404 returned error can't find the container with id ac34e39f904f6c3223122b05c3b47a7f54086789b49cca4feadd20aa88c04c74 Oct 30 21:40:18.818795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.818764 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:18.818909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.818831 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:18.818909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:18.818859 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:19.350500 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:19.350440 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" event={"ID":"bf7061e0-467e-4274-af65-b4485c807847","Type":"ContainerStarted","Data":"38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a"} Oct 30 21:40:19.350698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:19.350535 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" event={"ID":"bf7061e0-467e-4274-af65-b4485c807847","Type":"ContainerStarted","Data":"ac34e39f904f6c3223122b05c3b47a7f54086789b49cca4feadd20aa88c04c74"} Oct 30 21:40:19.352192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:19.352158 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" event={"ID":"7418a23c-fab4-4512-89d2-fad697a23192","Type":"ContainerStarted","Data":"c4fd8cce7ff54a5c0cdc857a35c15933f10dce0ffe62079659d4f47bcc8f645a"} Oct 30 21:40:19.352310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:19.352209 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" event={"ID":"7418a23c-fab4-4512-89d2-fad697a23192","Type":"ContainerStarted","Data":"c7c888c2ffe76869646bb3ad144e5f3b1b779be53eccfcbcf2e28cbc144723b6"} Oct 30 21:40:19.429909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:19.429001 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:19.432122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:19.431994 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:40:19.432122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:19.432055 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:40:19.484034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:19.483979 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podStartSLOduration=1.483958297 podStartE2EDuration="1.483958297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:19.477768563 +0000 UTC m=+1409.611141762" watchObservedRunningTime="2026-10-30 21:40:19.483958297 +0000 UTC m=+1409.617331499" Oct 30 21:40:20.362222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:20.362186 4526 generic.go:356] "Generic (PLEG): container finished" podID="7418a23c-fab4-4512-89d2-fad697a23192" containerID="c4fd8cce7ff54a5c0cdc857a35c15933f10dce0ffe62079659d4f47bcc8f645a" exitCode=0 Oct 30 21:40:20.362639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:20.362274 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" event={"ID":"7418a23c-fab4-4512-89d2-fad697a23192","Type":"ContainerDied","Data":"c4fd8cce7ff54a5c0cdc857a35c15933f10dce0ffe62079659d4f47bcc8f645a"} Oct 30 21:40:20.428393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:20.428364 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:40:20.428589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:20.428416 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:40:21.429242 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.429212 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:40:21.429661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.429264 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:40:21.564100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.564075 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:21.677657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.677570 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-openshift-ca\") pod \"7418a23c-fab4-4512-89d2-fad697a23192\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " Oct 30 21:40:21.677808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.677680 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7418a23c-fab4-4512-89d2-fad697a23192-kube-api-access-gwjdd\" (UniqueName: \"kubernetes.io/projected/7418a23c-fab4-4512-89d2-fad697a23192-kube-api-access-gwjdd\") pod \"7418a23c-fab4-4512-89d2-fad697a23192\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " Oct 30 21:40:21.677899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.677884 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-xks-ca\") pod \"7418a23c-fab4-4512-89d2-fad697a23192\" (UID: \"7418a23c-fab4-4512-89d2-fad697a23192\") " Oct 30 21:40:21.677953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.677931 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-openshift-ca" pod "7418a23c-fab4-4512-89d2-fad697a23192" (UID: "7418a23c-fab4-4512-89d2-fad697a23192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:21.678294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.678279 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:21.680281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.680238 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7418a23c-fab4-4512-89d2-fad697a23192-kube-api-access-gwjdd" pod "7418a23c-fab4-4512-89d2-fad697a23192" (UID: "7418a23c-fab4-4512-89d2-fad697a23192"). InnerVolumeSpecName "kube-api-access-gwjdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:21.680433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.680398 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-xks-ca" pod "7418a23c-fab4-4512-89d2-fad697a23192" (UID: "7418a23c-fab4-4512-89d2-fad697a23192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:21.779969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.779936 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7418a23c-fab4-4512-89d2-fad697a23192-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:21.779969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:21.779963 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwjdd\" (UniqueName: \"kubernetes.io/projected/7418a23c-fab4-4512-89d2-fad697a23192-kube-api-access-gwjdd\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:22.380384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:22.380353 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" Oct 30 21:40:22.380384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:22.380375 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n" event={"ID":"7418a23c-fab4-4512-89d2-fad697a23192","Type":"ContainerDied","Data":"c7c888c2ffe76869646bb3ad144e5f3b1b779be53eccfcbcf2e28cbc144723b6"} Oct 30 21:40:22.380608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:22.380424 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7c888c2ffe76869646bb3ad144e5f3b1b779be53eccfcbcf2e28cbc144723b6" Oct 30 21:40:22.429120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:22.429085 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:40:22.429305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:22.429139 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:40:23.428290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:23.428252 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:40:23.428551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:23.428318 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:40:24.429038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.428999 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:40:24.429469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.429062 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:40:24.908726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.908694 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t"] Oct 30 21:40:24.909398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.909377 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7418a23c-fab4-4512-89d2-fad697a23192" containerName="revoke-keys" Oct 30 21:40:24.909398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.909395 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7418a23c-fab4-4512-89d2-fad697a23192" containerName="revoke-keys" Oct 30 21:40:24.909667 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.909530 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7418a23c-fab4-4512-89d2-fad697a23192" containerName="revoke-keys" Oct 30 21:40:24.914618 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.914593 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:24.962807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.961528 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t"] Oct 30 21:40:24.990843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.990803 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97131b48-6654-45a4-a1f1-decdc0c60b58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t\" (UID: \"97131b48-6654-45a4-a1f1-decdc0c60b58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:24.991043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:24.990862 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkxg\" (UniqueName: \"kubernetes.io/projected/97131b48-6654-45a4-a1f1-decdc0c60b58-kube-api-access-fpkxg\") pod \"payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t\" (UID: \"97131b48-6654-45a4-a1f1-decdc0c60b58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:25.004201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.004162 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bed6793d\"" Oct 30 21:40:25.004201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.004182 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bed6793d-dockercfg-vwbsk\"" Oct 30 21:40:25.068236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.068202 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf"] Oct 30 21:40:25.073822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.073798 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:25.097766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.097727 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97131b48-6654-45a4-a1f1-decdc0c60b58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t\" (UID: \"97131b48-6654-45a4-a1f1-decdc0c60b58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:25.097968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.097797 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkxg\" (UniqueName: \"kubernetes.io/projected/97131b48-6654-45a4-a1f1-decdc0c60b58-kube-api-access-fpkxg\") pod \"payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t\" (UID: \"97131b48-6654-45a4-a1f1-decdc0c60b58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:25.097968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.097887 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9116af5-a82b-4c24-90f1-4f88ce419297-plugins-config-volume\") pod \"payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf\" (UID: \"b9116af5-a82b-4c24-90f1-4f88ce419297\") " pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:25.098169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.098150 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgnk\" (UniqueName: \"kubernetes.io/projected/b9116af5-a82b-4c24-90f1-4f88ce419297-kube-api-access-jsgnk\") pod \"payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf\" (UID: \"b9116af5-a82b-4c24-90f1-4f88ce419297\") " pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:25.101569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.099542 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97131b48-6654-45a4-a1f1-decdc0c60b58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t\" (UID: \"97131b48-6654-45a4-a1f1-decdc0c60b58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:25.143546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.141391 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf"] Oct 30 21:40:25.173664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.164091 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkxg\" (UniqueName: \"kubernetes.io/projected/97131b48-6654-45a4-a1f1-decdc0c60b58-kube-api-access-fpkxg\") pod \"payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t\" (UID: \"97131b48-6654-45a4-a1f1-decdc0c60b58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:25.208110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.208060 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgnk\" (UniqueName: \"kubernetes.io/projected/b9116af5-a82b-4c24-90f1-4f88ce419297-kube-api-access-jsgnk\") pod \"payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf\" (UID: \"b9116af5-a82b-4c24-90f1-4f88ce419297\") " pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:25.208327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.208278 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9116af5-a82b-4c24-90f1-4f88ce419297-plugins-config-volume\") pod \"payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf\" (UID: \"b9116af5-a82b-4c24-90f1-4f88ce419297\") " pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:25.209314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.209288 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9116af5-a82b-4c24-90f1-4f88ce419297-plugins-config-volume\") pod \"payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf\" (UID: \"b9116af5-a82b-4c24-90f1-4f88ce419297\") " pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:25.231121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.231086 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:25.267031 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.266969 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgnk\" (UniqueName: \"kubernetes.io/projected/b9116af5-a82b-4c24-90f1-4f88ce419297-kube-api-access-jsgnk\") pod \"payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf\" (UID: \"b9116af5-a82b-4c24-90f1-4f88ce419297\") " pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:25.400819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.400783 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:25.434574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.431414 4526 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:40:25.434574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.431471 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 21:40:25.614941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.614907 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m"] Oct 30 21:40:25.620612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.620582 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.625420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.625389 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.625590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.625571 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58prd\" (UniqueName: \"kubernetes.io/projected/d141cab6-5b5b-4a79-bfc4-126ae900563e-kube-api-access-58prd\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.625661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.625648 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.668874 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:25.668838 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97131b48_6654_45a4_a1f1_decdc0c60b58.slice/crio-f0dbf821f8bf20842b25a222d265cd848afe6b2477cc26a02905641d560d1c22 WatchSource:0}: Error finding container f0dbf821f8bf20842b25a222d265cd848afe6b2477cc26a02905641d560d1c22: Status 404 returned error can't find the container with id f0dbf821f8bf20842b25a222d265cd848afe6b2477cc26a02905641d560d1c22 Oct 30 21:40:25.670019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.669994 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t"] Oct 30 21:40:25.673812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.672346 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:40:25.673812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.672589 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:40:25.673812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.673781 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:40:25.674128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.674094 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:40:25.677388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.677360 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m"] Oct 30 21:40:25.726696 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.726659 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.726865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.726792 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58prd\" (UniqueName: \"kubernetes.io/projected/d141cab6-5b5b-4a79-bfc4-126ae900563e-kube-api-access-58prd\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.726946 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.726872 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.727961 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.727919 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.729442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.729415 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.784936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.784895 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58prd\" (UniqueName: \"kubernetes.io/projected/d141cab6-5b5b-4a79-bfc4-126ae900563e-kube-api-access-58prd\") pod \"maas-api-revoke-sub-ae7eab26-vff6m\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:25.795536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.795506 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf"] Oct 30 21:40:25.795721 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:25.795696 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9116af5_a82b_4c24_90f1_4f88ce419297.slice/crio-b68f4c7c5cab67e1980d5f3cf9b13b7881e8b87a00b272452990453c8ed3c22e WatchSource:0}: Error finding container b68f4c7c5cab67e1980d5f3cf9b13b7881e8b87a00b272452990453c8ed3c22e: Status 404 returned error can't find the container with id b68f4c7c5cab67e1980d5f3cf9b13b7881e8b87a00b272452990453c8ed3c22e Oct 30 21:40:25.936635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:25.936592 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:26.339954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.339919 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m"] Oct 30 21:40:26.340193 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:26.340169 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd141cab6_5b5b_4a79_bfc4_126ae900563e.slice/crio-3451735ee7dda54d655dbcea7576f21152bffcd44a57a5fbb1687336932adb1f WatchSource:0}: Error finding container 3451735ee7dda54d655dbcea7576f21152bffcd44a57a5fbb1687336932adb1f: Status 404 returned error can't find the container with id 3451735ee7dda54d655dbcea7576f21152bffcd44a57a5fbb1687336932adb1f Oct 30 21:40:26.433601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.433571 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:26.434036 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.434013 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:26.434665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.434645 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:40:26.448472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.448444 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" event={"ID":"b9116af5-a82b-4c24-90f1-4f88ce419297","Type":"ContainerStarted","Data":"1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748"} Oct 30 21:40:26.448654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.448507 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" event={"ID":"b9116af5-a82b-4c24-90f1-4f88ce419297","Type":"ContainerStarted","Data":"b68f4c7c5cab67e1980d5f3cf9b13b7881e8b87a00b272452990453c8ed3c22e"} Oct 30 21:40:26.448654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.448538 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:26.449942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.449911 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" event={"ID":"d141cab6-5b5b-4a79-bfc4-126ae900563e","Type":"ContainerStarted","Data":"3451735ee7dda54d655dbcea7576f21152bffcd44a57a5fbb1687336932adb1f"} Oct 30 21:40:26.451432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.451409 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" event={"ID":"97131b48-6654-45a4-a1f1-decdc0c60b58","Type":"ContainerStarted","Data":"faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd"} Oct 30 21:40:26.451562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.451453 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" event={"ID":"97131b48-6654-45a4-a1f1-decdc0c60b58","Type":"ContainerStarted","Data":"f0dbf821f8bf20842b25a222d265cd848afe6b2477cc26a02905641d560d1c22"} Oct 30 21:40:26.451650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.451636 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:26.666405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.666352 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" podStartSLOduration=2.666337155 podStartE2EDuration="2.666337155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:26.662922914 +0000 UTC m=+1416.796296112" watchObservedRunningTime="2026-10-30 21:40:26.666337155 +0000 UTC m=+1416.799710353" Oct 30 21:40:26.897468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:26.897357 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" podStartSLOduration=1.897337942 podStartE2EDuration="1.897337942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:26.895815184 +0000 UTC m=+1417.029188382" watchObservedRunningTime="2026-10-30 21:40:26.897337942 +0000 UTC m=+1417.030711151" Oct 30 21:40:27.449191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.449152 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm"] Oct 30 21:40:27.453857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.453834 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:27.474305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.474270 4526 generic.go:356] "Generic (PLEG): container finished" podID="d141cab6-5b5b-4a79-bfc4-126ae900563e" containerID="bf9051bc35f51af9b332de093307661c182757a78d7d2dbfc2b425391fc9a467" exitCode=0 Oct 30 21:40:27.474534 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.474423 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" event={"ID":"d141cab6-5b5b-4a79-bfc4-126ae900563e","Type":"ContainerDied","Data":"bf9051bc35f51af9b332de093307661c182757a78d7d2dbfc2b425391fc9a467"} Oct 30 21:40:27.484294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.484251 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8h7\" (UniqueName: \"kubernetes.io/projected/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-kube-api-access-5t8h7\") pod \"payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm\" (UID: \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:27.484471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.484323 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm\" (UID: \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:27.508370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.508332 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-385396af\"" Oct 30 21:40:27.508581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.508538 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-385396af-dockercfg-szntk\"" Oct 30 21:40:27.517272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.517229 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm"] Oct 30 21:40:27.597874 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.597770 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8h7\" (UniqueName: \"kubernetes.io/projected/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-kube-api-access-5t8h7\") pod \"payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm\" (UID: \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:27.598133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.598114 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm\" (UID: \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:27.599227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.599181 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm\" (UID: \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:27.663783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.663737 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8h7\" (UniqueName: \"kubernetes.io/projected/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-kube-api-access-5t8h7\") pod \"payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm\" (UID: \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:27.781287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:27.781258 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:28.167608 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:28.167563 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a65c7c3_4ef5_4041_aa48_d9de167baf8e.slice/crio-19fb0ac11edaef8e75d91f0bbd784d0e76d22f5eb584a2f6db74408f1ce15191 WatchSource:0}: Error finding container 19fb0ac11edaef8e75d91f0bbd784d0e76d22f5eb584a2f6db74408f1ce15191: Status 404 returned error can't find the container with id 19fb0ac11edaef8e75d91f0bbd784d0e76d22f5eb584a2f6db74408f1ce15191 Oct 30 21:40:28.168832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.168810 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm"] Oct 30 21:40:28.485114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.485025 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" event={"ID":"7a65c7c3-4ef5-4041-aa48-d9de167baf8e","Type":"ContainerStarted","Data":"c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8"} Oct 30 21:40:28.485114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.485087 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" event={"ID":"7a65c7c3-4ef5-4041-aa48-d9de167baf8e","Type":"ContainerStarted","Data":"19fb0ac11edaef8e75d91f0bbd784d0e76d22f5eb584a2f6db74408f1ce15191"} Oct 30 21:40:28.485633 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.485118 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:28.604648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.604586 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" podStartSLOduration=1.6045714960000002 podStartE2EDuration="1.604571496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:28.602527973 +0000 UTC m=+1418.735901172" watchObservedRunningTime="2026-10-30 21:40:28.604571496 +0000 UTC m=+1418.737944693" Oct 30 21:40:28.648271 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.648246 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:28.723265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.723230 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d141cab6-5b5b-4a79-bfc4-126ae900563e-kube-api-access-58prd\" (UniqueName: \"kubernetes.io/projected/d141cab6-5b5b-4a79-bfc4-126ae900563e-kube-api-access-58prd\") pod \"d141cab6-5b5b-4a79-bfc4-126ae900563e\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " Oct 30 21:40:28.723595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.723473 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-openshift-ca\") pod \"d141cab6-5b5b-4a79-bfc4-126ae900563e\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " Oct 30 21:40:28.725729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.723590 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-xks-ca\") pod \"d141cab6-5b5b-4a79-bfc4-126ae900563e\" (UID: \"d141cab6-5b5b-4a79-bfc4-126ae900563e\") " Oct 30 21:40:28.728187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.727269 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-openshift-ca" pod "d141cab6-5b5b-4a79-bfc4-126ae900563e" (UID: "d141cab6-5b5b-4a79-bfc4-126ae900563e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:28.728187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.727987 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d141cab6-5b5b-4a79-bfc4-126ae900563e-kube-api-access-58prd" pod "d141cab6-5b5b-4a79-bfc4-126ae900563e" (UID: "d141cab6-5b5b-4a79-bfc4-126ae900563e"). InnerVolumeSpecName "kube-api-access-58prd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:28.728777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.728731 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-xks-ca" pod "d141cab6-5b5b-4a79-bfc4-126ae900563e" (UID: "d141cab6-5b5b-4a79-bfc4-126ae900563e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:28.825995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.825963 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.825995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.825997 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d141cab6-5b5b-4a79-bfc4-126ae900563e-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:28.826212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:28.826008 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58prd\" (UniqueName: \"kubernetes.io/projected/d141cab6-5b5b-4a79-bfc4-126ae900563e-kube-api-access-58prd\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:40:29.493692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:29.493662 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" Oct 30 21:40:29.494107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:29.493688 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m" event={"ID":"d141cab6-5b5b-4a79-bfc4-126ae900563e","Type":"ContainerDied","Data":"3451735ee7dda54d655dbcea7576f21152bffcd44a57a5fbb1687336932adb1f"} Oct 30 21:40:29.494107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:29.493738 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3451735ee7dda54d655dbcea7576f21152bffcd44a57a5fbb1687336932adb1f" Oct 30 21:40:34.319516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.318625 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:40:34.431508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.428651 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.481750 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.481647 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.571723 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.571609 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.572398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.572357 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:40:34.686839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.686792 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.786245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.784999 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.788087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.788033 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.852502 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.852356 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": context deadline exceeded" Oct 30 21:40:34.886721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.886657 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.980468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:34.980428 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.071522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.064609 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.117576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.107268 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.219515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.213994 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 30 21:40:35.326515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.323600 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.326515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.324428 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:40:35.430578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.430314 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 30 21:40:35.434520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.430924 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:40:35.566516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.545835 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.566516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.547026 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:40:35.749575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.572557 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.749575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.573037 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:40:35.749575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.574567 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.749575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.574929 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:40:35.749575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.690291 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.749575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.690817 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:40:35.829519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.816672 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.829519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.818026 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:40:35.880515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.831663 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:40:35.880515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.832073 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:40:35.942517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.924656 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.942517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:35.925597 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:40:36.024596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.023605 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:40:36.066225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.065574 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.129542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.126136 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:36.129542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.126201 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.136856 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.132133 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.313661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.288041 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.396649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.396598 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.599737 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.436907 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.704237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.704189 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.810592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.705569 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.812882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.812843 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.855637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.824926 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 21:40:36.855637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.825601 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.855637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.825675 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.894898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:36.894836 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.047516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.022916 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:40:37.060766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.060697 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:40:37.205574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.176981 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.332685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.182284 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:40:37.332685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.223574 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 21:40:37.604332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.604270 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:40:37.610854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.604892 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.835634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.835584 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.861111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.836133 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 21:40:37.895515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.836169 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 30 21:40:37.897429 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.836213 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.923995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.836233 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Oct 30 21:40:37.924233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:37.836250 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 21:40:38.061648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.061596 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.085465 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:40:38.085429 4526 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.104s" Oct 30 21:40:38.142954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.134889 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.295441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.295399 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.297090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.297052 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 21:40:38.297719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.297696 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:40:38.588514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.562367 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:40:38.595672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.595640 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:40:38.691682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.691655 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:40:38.822519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.818689 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:40:38.822519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.818767 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:40:38.822519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.611605 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded" Oct 30 21:40:38.822519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.611638 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 21:40:38.896744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.871234 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.897524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.897455 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.964677 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.964615 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 21:40:39.077519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.964835 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" podUID="c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 30 21:40:39.077519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:38.964875 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 21:40:39.140515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.081076 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 21:40:39.164791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.155031 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 21:40:39.319530 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.319466 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:39.320766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.320731 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:39.322800 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.322769 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 21:40:39.356543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.350259 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vns6g" podUID="cdf20d6e-28cb-40ef-ba2b-b2108efbe6da" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.19:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:39.459028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.458938 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:40:39.498116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.498087 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 21:40:39.519766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.519739 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf"] Oct 30 21:40:39.539441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.539305 4526 cpu_manager.go:414] "containerMap: removing container" podUID="d141cab6-5b5b-4a79-bfc4-126ae900563e" containerName="revoke-keys" Oct 30 21:40:39.539992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.539975 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d141cab6-5b5b-4a79-bfc4-126ae900563e" containerName="revoke-keys" Oct 30 21:40:39.540520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.540365 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="d141cab6-5b5b-4a79-bfc4-126ae900563e" containerName="revoke-keys" Oct 30 21:40:39.563792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.563759 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 21:40:39.563962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.563843 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:40:39.564032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.563962 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:40:39.565199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.565173 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.600251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.600221 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8"] Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.629944 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.629946 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmthw\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-kube-api-access-hmthw\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.630257 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.630292 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8"] Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.630330 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.630393 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.630560 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.630689 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.630803 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.630860 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:40:39.631582 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.631125 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 21:40:39.636917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.632699 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.636917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.634361 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.636917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.636249 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf"] Oct 30 21:40:39.636917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.636404 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:40:39.638121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.637975 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a502da11-e60f-46c3-9d00-22c05e102c21-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.638121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.638047 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a502da11-e60f-46c3-9d00-22c05e102c21-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.638312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.638236 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.638373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.638312 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.646223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.646199 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 21:40:39.739729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.739651 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmthw\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-kube-api-access-hmthw\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.739729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.739723 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.740257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.739954 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.740327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740299 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.740378 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740342 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.740521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740501 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.740705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740680 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.740767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740719 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.740767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740745 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.740836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740815 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.740880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740864 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.740991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.740968 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scchl\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-kube-api-access-scchl\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.741071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741045 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.741176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741153 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a502da11-e60f-46c3-9d00-22c05e102c21-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.741250 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741170 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.741314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741243 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a502da11-e60f-46c3-9d00-22c05e102c21-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.741314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741274 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.741406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741328 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.741741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741718 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.742030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741749 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.742030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.741805 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.742030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.742004 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a502da11-e60f-46c3-9d00-22c05e102c21-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.742202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.742134 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.744009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.743987 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.744105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.744028 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a502da11-e60f-46c3-9d00-22c05e102c21-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.796836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.796799 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.797308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.797283 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmthw\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-kube-api-access-hmthw\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.843336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843304 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.843575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843354 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scchl\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-kube-api-access-scchl\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.843575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843469 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.843575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843549 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.843763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843692 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.843817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843766 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.843874 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843834 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.843931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843870 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.843931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843895 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.844040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.843938 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.844115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.844095 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.844173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.844142 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.844229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.844192 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.844298 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.844278 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.846232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.846215 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.846614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.846594 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.900912 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.900876 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.901063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.900949 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scchl\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-kube-api-access-scchl\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:39.964054 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.964015 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:39.967133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:39.967100 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:40.408341 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:40.408289 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda502da11_e60f_46c3_9d00_22c05e102c21.slice/crio-327ddcb29b16c22fafe1674ac8b6cd0015558908e19fea0bd274b79177f57a84 WatchSource:0}: Error finding container 327ddcb29b16c22fafe1674ac8b6cd0015558908e19fea0bd274b79177f57a84: Status 404 returned error can't find the container with id 327ddcb29b16c22fafe1674ac8b6cd0015558908e19fea0bd274b79177f57a84 Oct 30 21:40:40.410071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:40.410045 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf"] Oct 30 21:40:40.410210 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:40:40.410192 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb262a6c9_4ea9_4b7f_9e4d_c1fa65d196da.slice/crio-5e2ce7da469b351e1ea67df61d37be9d32c1af9bd7c2cb1241b9997d74096738 WatchSource:0}: Error finding container 5e2ce7da469b351e1ea67df61d37be9d32c1af9bd7c2cb1241b9997d74096738: Status 404 returned error can't find the container with id 5e2ce7da469b351e1ea67df61d37be9d32c1af9bd7c2cb1241b9997d74096738 Oct 30 21:40:40.411384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:40.411345 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:40.411506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:40.411423 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:40.411506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:40.411455 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:40.413104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:40.413075 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:40.413204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:40.413135 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:40.413204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:40.413164 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:40.415705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:40.415684 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8"] Oct 30 21:40:41.463112 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.463071 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" event={"ID":"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da","Type":"ContainerStarted","Data":"1257e621b3e782f32dcbde87e10c305b07af8fb00416fd674d64f0e5e81e6c6a"} Oct 30 21:40:41.463845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.463796 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" event={"ID":"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da","Type":"ContainerStarted","Data":"5e2ce7da469b351e1ea67df61d37be9d32c1af9bd7c2cb1241b9997d74096738"} Oct 30 21:40:41.465971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.465925 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" event={"ID":"a502da11-e60f-46c3-9d00-22c05e102c21","Type":"ContainerStarted","Data":"6161c51f505c1483c44be155e91d46786b2ddf0263917ce8a37afbd66dfba50e"} Oct 30 21:40:41.466076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.465973 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" event={"ID":"a502da11-e60f-46c3-9d00-22c05e102c21","Type":"ContainerStarted","Data":"327ddcb29b16c22fafe1674ac8b6cd0015558908e19fea0bd274b79177f57a84"} Oct 30 21:40:41.661674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.661605 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" podStartSLOduration=5.661580826 podStartE2EDuration="5.661580826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:41.632884654 +0000 UTC m=+1431.766257855" watchObservedRunningTime="2026-10-30 21:40:41.661580826 +0000 UTC m=+1431.794954025" Oct 30 21:40:41.757123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.757002 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" podStartSLOduration=5.756979307 podStartE2EDuration="5.756979307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:41.754050992 +0000 UTC m=+1431.887424191" watchObservedRunningTime="2026-10-30 21:40:41.756979307 +0000 UTC m=+1431.890352505" Oct 30 21:40:41.965135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.965097 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:41.966775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.966743 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:40:41.966947 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.966811 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:40:41.967392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.967370 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:41.968232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.968206 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:40:41.968338 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:41.968248 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:40:42.481334 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:40:42.481299 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:40:42.965344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:42.965310 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:40:42.965570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:42.965361 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:40:42.967683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:42.967654 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:40:42.967793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:42.967706 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:40:43.964640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:43.964600 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:40:43.965122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:43.964658 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:40:43.967419 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:43.967393 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:40:43.967473 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:43.967442 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:40:44.965301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:44.965261 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:40:44.965748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:44.965326 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:40:44.967593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:44.967572 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:40:44.967643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:44.967613 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:40:45.965190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:45.965154 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:40:45.965381 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:45.965218 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:40:45.968002 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:45.967978 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:40:45.968080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:45.968041 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:40:46.965300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:46.965261 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 21:40:46.965483 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:46.965322 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 21:40:46.967596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:46.967570 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:40:46.967710 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:46.967621 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:40:48.136124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:48.136089 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:48.136638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:48.136603 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:48.137839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:48.137819 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:40:48.186104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:48.185095 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:48.186104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:48.185440 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:48.186613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:48.186591 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:40:50.475398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.475357 4526 scope.go:122] "RemoveContainer" containerID="c9520b3ed8fe0a035bbe81e96238fcda28c39b25ea2f91505c136cfd157d0d0c" Oct 30 21:40:50.486510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.486470 4526 scope.go:122] "RemoveContainer" containerID="f33df8ee237feedb22639768b2eb34bf847351e3ff244ffd5af5b603503739b9" Oct 30 21:40:50.502179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.502159 4526 scope.go:122] "RemoveContainer" containerID="dd554a15dbd5ae850fdeb3f108d53e0d632877de933fafe6fa77a1362af75056" Oct 30 21:40:50.512673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.512652 4526 scope.go:122] "RemoveContainer" containerID="4f1aa7682512dd32aa04bbb7f350b9073aad00198dd078f807d34e10d57dc338" Oct 30 21:40:50.522337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.522319 4526 scope.go:122] "RemoveContainer" containerID="d2ff5addbb7d5a6885820eb60fdcdea72bfa3b312131d47ce967a1ff6302ef0f" Oct 30 21:40:50.532151 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.532129 4526 scope.go:122] "RemoveContainer" containerID="71ed0323d530f9a0ac6c21bff5e1b1fa74f9b2b7576793626d5a8435e64f73e8" Oct 30 21:40:50.542018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.541999 4526 scope.go:122] "RemoveContainer" containerID="6208a37b3ca74027b37972badf099f7d960d0ce19039d982bf8d6b10440bdabe" Oct 30 21:40:50.552111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.552088 4526 scope.go:122] "RemoveContainer" containerID="d892958e2f326deddf671917080fd9e6b2a77f4b0ee944ba5c07cc7c3d0b1ca9" Oct 30 21:40:50.561663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:40:50.561639 4526 scope.go:122] "RemoveContainer" containerID="92cbcab3e8c44661c62f1cadcceb24f2117015c9b847b647853d7ebeb17c90c5" Oct 30 21:41:03.260723 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.260691 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c"] Oct 30 21:41:03.267225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.267204 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.312234 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.312202 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks"] Oct 30 21:41:03.316764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.316745 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.343214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343185 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.343367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343227 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.343367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343351 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.343443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343410 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.343535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343515 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.343587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343567 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.343692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343678 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l526r\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-kube-api-access-l526r\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.343731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343720 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.343806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.343793 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.369136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.369105 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c"] Oct 30 21:41:03.445140 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445097 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.445348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445187 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.445348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445216 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.445348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445265 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l526r\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-kube-api-access-l526r\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.445604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445399 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.445604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445499 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.445604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445551 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.445604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445579 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.445821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445608 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.445879 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445834 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.446013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445992 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.446130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446034 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddvv5\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-kube-api-access-ddvv5\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.446130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446109 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.446291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446144 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.446291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446222 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.446291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.445998 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.446463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446292 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.446463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446424 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.446630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446462 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.446630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446561 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500581f8-6b15-45f0-b21d-37ef93d26b53-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.446630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446595 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.446992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.446969 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.447221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.447202 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.448689 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.448667 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.449183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.449162 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.475137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.475093 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks"] Oct 30 21:41:03.502535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.502476 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l526r\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-kube-api-access-l526r\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.502840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.502818 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.547791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.547758 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddvv5\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-kube-api-access-ddvv5\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.547963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.547812 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.547963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.547925 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.547984 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548019 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500581f8-6b15-45f0-b21d-37ef93d26b53-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548087 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548104 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548173 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548234 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548210 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548255 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548313 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548277 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548441 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548674 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.548824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.548738 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500581f8-6b15-45f0-b21d-37ef93d26b53-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.550423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.550406 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.550845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.550823 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.581244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.581205 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:03.604553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.604527 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.604887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.604869 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddvv5\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-kube-api-access-ddvv5\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:03.632702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:03.632671 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:04.015798 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:41:04.015769 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb216ab_5a92_4328_8538_0c7c1e2d8788.slice/crio-e9ad351e3cb256c23930e5fc24c5ccf97716d01be875d675b2849a9d43ad6608 WatchSource:0}: Error finding container e9ad351e3cb256c23930e5fc24c5ccf97716d01be875d675b2849a9d43ad6608: Status 404 returned error can't find the container with id e9ad351e3cb256c23930e5fc24c5ccf97716d01be875d675b2849a9d43ad6608 Oct 30 21:41:04.021053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.017683 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c"] Oct 30 21:41:04.021053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.020853 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.021053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.020932 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.021053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.020973 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.069671 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.069641 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks"] Oct 30 21:41:04.078040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.077965 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" event={"ID":"3fb216ab-5a92-4328-8538-0c7c1e2d8788","Type":"ContainerStarted","Data":"e9ad351e3cb256c23930e5fc24c5ccf97716d01be875d675b2849a9d43ad6608"} Oct 30 21:41:04.078821 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:41:04.078792 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500581f8_6b15_45f0_b21d_37ef93d26b53.slice/crio-8ae6e353d92473dcad767ee979f4ee27eb34753d794fbe6c70fa74b5111a4e8f WatchSource:0}: Error finding container 8ae6e353d92473dcad767ee979f4ee27eb34753d794fbe6c70fa74b5111a4e8f: Status 404 returned error can't find the container with id 8ae6e353d92473dcad767ee979f4ee27eb34753d794fbe6c70fa74b5111a4e8f Oct 30 21:41:04.081736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.081709 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.081821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.081776 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.081821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:04.081808 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:05.088235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.088200 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" event={"ID":"500581f8-6b15-45f0-b21d-37ef93d26b53","Type":"ContainerStarted","Data":"204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a"} Oct 30 21:41:05.088681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.088255 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" event={"ID":"500581f8-6b15-45f0-b21d-37ef93d26b53","Type":"ContainerStarted","Data":"8ae6e353d92473dcad767ee979f4ee27eb34753d794fbe6c70fa74b5111a4e8f"} Oct 30 21:41:05.090084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.090060 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" event={"ID":"3fb216ab-5a92-4328-8538-0c7c1e2d8788","Type":"ContainerStarted","Data":"0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61"} Oct 30 21:41:05.284411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.284359 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" podStartSLOduration=2.284346535 podStartE2EDuration="2.284346535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:05.283003537 +0000 UTC m=+1455.416376734" watchObservedRunningTime="2026-10-30 21:41:05.284346535 +0000 UTC m=+1455.417719732" Oct 30 21:41:05.450282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.449602 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" podStartSLOduration=2.449577584 podStartE2EDuration="2.449577584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:05.43445457 +0000 UTC m=+1455.567827770" watchObservedRunningTime="2026-10-30 21:41:05.449577584 +0000 UTC m=+1455.582950783" Oct 30 21:41:05.582332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.582302 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:05.584286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.584259 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:41:05.584406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.584311 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:41:05.641511 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.640847 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:05.644530 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.644218 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:41:05.644530 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:05.644282 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:41:06.582635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:06.582597 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:41:06.583124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:06.583084 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:41:06.644360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:06.643895 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:41:06.644360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:06.643957 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:41:07.583857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:07.583823 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:41:07.584569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:07.584465 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:41:07.640454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:07.640423 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:41:07.640758 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:07.640730 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:41:08.582540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:08.582479 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:41:08.582699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:08.582563 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:41:08.635247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:08.635212 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:41:08.636341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:08.636308 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:41:09.582033 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:09.581996 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:41:09.582241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:09.582047 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:41:09.647516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:09.645245 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:41:09.647516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:09.645324 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:41:10.582545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:10.582507 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.582725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:10.582566 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:41:10.633553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:10.633505 4526 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.633751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:10.633613 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 21:41:11.742811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:11.742781 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:11.743294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:11.743275 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:11.744314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:11.744295 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:41:11.757761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:11.757737 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:11.758048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:11.758026 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:11.758900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:11.758881 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:41:12.480402 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:41:12.480366 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:41:36.652785 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.652687 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj"] Oct 30 21:41:36.658770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.658750 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.749897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.749858 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.750080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.749999 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.750136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.750120 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.750197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.750181 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.750243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.750231 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.750347 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.750333 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5703ef72-0530-4cc8-929b-36a65f3b55a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.750401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.750389 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.750445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.750410 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.750517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.750480 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpgg\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-kube-api-access-rcpgg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.851943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.851913 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.851964 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852014 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852036 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852077 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852143 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5703ef72-0530-4cc8-929b-36a65f3b55a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852187 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852213 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852297 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpgg\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-kube-api-access-rcpgg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852314 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852624 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852584 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852800 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852775 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.852885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.852866 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.853070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.853052 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5703ef72-0530-4cc8-929b-36a65f3b55a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.855310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.855284 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.855516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.855471 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.874569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.874535 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj"] Oct 30 21:41:36.921603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.921535 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.921778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.921755 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpgg\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-kube-api-access-rcpgg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hqlmj\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:36.974279 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:36.974245 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:37.435813 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:41:37.435784 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5703ef72_0530_4cc8_929b_36a65f3b55a5.slice/crio-2fb28d9b6f4d56771d91d8fa44ee3eb54d7bc738bf3440a990eb9a4b8a32ceee WatchSource:0}: Error finding container 2fb28d9b6f4d56771d91d8fa44ee3eb54d7bc738bf3440a990eb9a4b8a32ceee: Status 404 returned error can't find the container with id 2fb28d9b6f4d56771d91d8fa44ee3eb54d7bc738bf3440a990eb9a4b8a32ceee Oct 30 21:41:37.437894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:37.437871 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj"] Oct 30 21:41:37.438874 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:37.438846 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:37.438938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:37.438900 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:37.438938 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:37.438931 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:37.526744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:37.516060 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" event={"ID":"5703ef72-0530-4cc8-929b-36a65f3b55a5","Type":"ContainerStarted","Data":"2fb28d9b6f4d56771d91d8fa44ee3eb54d7bc738bf3440a990eb9a4b8a32ceee"} Oct 30 21:41:38.535403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:38.535364 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" event={"ID":"5703ef72-0530-4cc8-929b-36a65f3b55a5","Type":"ContainerStarted","Data":"47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54"} Oct 30 21:41:38.728575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:38.728509 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podStartSLOduration=2.7284718100000003 podStartE2EDuration="2.72847181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:38.711871504 +0000 UTC m=+1488.845244702" watchObservedRunningTime="2026-10-30 21:41:38.72847181 +0000 UTC m=+1488.861845012" Oct 30 21:41:38.975393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:38.975299 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:38.977355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:38.977319 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.977518 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:38.977383 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:41:39.975696 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:39.975661 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.976128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:39.975726 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:41:40.480832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.480785 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm"] Oct 30 21:41:40.485445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.485419 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.532606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.532571 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.532785 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.532685 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff3077f-ff81-4f97-b62a-27618625a9b2-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.532899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.532877 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.532937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.532910 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.532993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.532981 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.533027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.533010 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.533081 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.533065 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.533158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.533143 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6pw\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-kube-api-access-6m6pw\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.533192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.533177 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.537587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.537560 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bed6793d-openshift-default-dockercfg-6ldgq\"" Oct 30 21:41:40.629956 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.626187 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm"] Oct 30 21:41:40.639229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.638954 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.639229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.639009 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.639229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.639063 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.639229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.639100 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.639229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.639157 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.639229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.639207 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6pw\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-kube-api-access-6m6pw\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.639715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.639240 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.639715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.639384 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.639715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.639436 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff3077f-ff81-4f97-b62a-27618625a9b2-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.649708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.648240 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.649708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.649637 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff3077f-ff81-4f97-b62a-27618625a9b2-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.650305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.650277 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.650557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.650532 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.650803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.650782 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.653729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.653703 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.656818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.656717 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.726052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.723245 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6pw\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-kube-api-access-6m6pw\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.726052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.725842 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-424pm\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.827541 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.827277 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:40.974666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.974632 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.974884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:40.974695 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:41:41.293030 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:41:41.293002 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff3077f_ff81_4f97_b62a_27618625a9b2.slice/crio-e314e8e88fdda24ae440c26bea9cca4b9a153b500f116a214ece86de2c4d788e WatchSource:0}: Error finding container e314e8e88fdda24ae440c26bea9cca4b9a153b500f116a214ece86de2c4d788e: Status 404 returned error can't find the container with id e314e8e88fdda24ae440c26bea9cca4b9a153b500f116a214ece86de2c4d788e Oct 30 21:41:41.293436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.293412 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm"] Oct 30 21:41:41.320579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.320545 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:41.320724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.320627 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:41.320724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.320663 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:41.565965 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.565872 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" event={"ID":"fff3077f-ff81-4f97-b62a-27618625a9b2","Type":"ContainerStarted","Data":"dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682"} Oct 30 21:41:41.565965 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.565947 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" event={"ID":"fff3077f-ff81-4f97-b62a-27618625a9b2","Type":"ContainerStarted","Data":"e314e8e88fdda24ae440c26bea9cca4b9a153b500f116a214ece86de2c4d788e"} Oct 30 21:41:41.685745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.685675 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" podStartSLOduration=1.685654539 podStartE2EDuration="1.685654539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:41.684087085 +0000 UTC m=+1491.817460284" watchObservedRunningTime="2026-10-30 21:41:41.685654539 +0000 UTC m=+1491.819027738" Oct 30 21:41:41.829653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.828324 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:41.974854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.974816 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:41:41.975055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:41.974879 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:41:42.485761 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:41:42.485722 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:41:42.576971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.576927 4526 generic.go:356] "Generic (PLEG): container finished" podID="0838b810-8246-4b65-a450-cc33e4cc1121" containerID="e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed" exitCode=0 Oct 30 21:41:42.577150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.576996 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" event={"ID":"0838b810-8246-4b65-a450-cc33e4cc1121","Type":"ContainerDied","Data":"e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed"} Oct 30 21:41:42.577450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.577436 4526 scope.go:122] "RemoveContainer" containerID="e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed" Oct 30 21:41:42.579449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.579418 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:42.579619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.579545 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:42.579619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.579607 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:42.856368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.855595 4526 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-424pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.856368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.855666 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 21:41:42.975356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.975255 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.975356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:42.975330 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:41:43.592426 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:43.592390 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" event={"ID":"0838b810-8246-4b65-a450-cc33e4cc1121","Type":"ContainerStarted","Data":"e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23"} Oct 30 21:41:43.835837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:43.835801 4526 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-424pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:41:43.836081 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:43.836056 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 21:41:43.975574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:43.974635 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:41:43.975574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:43.974702 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:41:44.563050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:44.563010 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:41:44.564262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:44.564233 4526 patch_prober.go:29] interesting pod/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:41:44.564366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:44.564296 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:41:44.830572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:44.829716 4526 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-424pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:41:44.830572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:44.829779 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 21:41:44.974805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:44.974770 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:41:44.974989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:44.974821 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:41:45.292238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:45.292204 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:41:45.563397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:45.563307 4526 patch_prober.go:29] interesting pod/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:41:45.563397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:45.563362 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:41:45.851571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:45.843479 4526 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-424pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:41:45.851571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:45.843556 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 21:41:45.975150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:45.975057 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 21:41:45.975150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:45.975126 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 21:41:46.563810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:46.563639 4526 patch_prober.go:29] interesting pod/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:41:46.563810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:46.563712 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:41:46.832571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:46.830782 4526 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-424pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:41:46.832571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:46.830844 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 21:41:47.569188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:47.569150 4526 patch_prober.go:29] interesting pod/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:41:47.569885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:47.569856 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:41:47.681525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:47.681201 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:47.682614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:47.682589 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:47.685068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:47.685048 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 21:41:47.828572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:47.828037 4526 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-424pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:41:47.828572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:47.828098 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 21:41:48.562880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:48.562837 4526 patch_prober.go:29] interesting pod/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:41:48.563106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:48.562897 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 30 21:41:48.851110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:48.851017 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:48.851638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:48.851425 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:48.852149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:48.852127 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:41:49.568281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:49.568243 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:41:49.569573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:41:49.569548 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:42:04.863887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:04.863850 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb"] Oct 30 21:42:04.870203 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:04.870176 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:04.924708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:04.924677 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb"] Oct 30 21:42:04.967272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:04.967226 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:42:04.967751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:04.967282 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:42:05.000296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.000265 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:42:05.002104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.002080 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.002251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.002115 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8q2m\" (UniqueName: \"kubernetes.io/projected/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-kube-api-access-f8q2m\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.002337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.002321 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.016740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.016711 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:42:05.103832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.103798 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8q2m\" (UniqueName: \"kubernetes.io/projected/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-kube-api-access-f8q2m\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.104017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.103938 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.104138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.104126 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.104748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.104725 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.106675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.106654 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.160379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.160302 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8q2m\" (UniqueName: \"kubernetes.io/projected/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-kube-api-access-f8q2m\") pod \"maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.185435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.185401 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:05.569682 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:42:05.569644 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3ac6b8_d3ec_48e6_90e2_ac87e6527fe3.slice/crio-e1376d53de871fcbba60a8d1b3cc79cfc2ba2904e05fad0ef8e7879718d5d32c WatchSource:0}: Error finding container e1376d53de871fcbba60a8d1b3cc79cfc2ba2904e05fad0ef8e7879718d5d32c: Status 404 returned error can't find the container with id e1376d53de871fcbba60a8d1b3cc79cfc2ba2904e05fad0ef8e7879718d5d32c Oct 30 21:42:05.571057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.571033 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb"] Oct 30 21:42:05.924461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:05.924426 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" event={"ID":"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3","Type":"ContainerStarted","Data":"e1376d53de871fcbba60a8d1b3cc79cfc2ba2904e05fad0ef8e7879718d5d32c"} Oct 30 21:42:06.934282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:06.934183 4526 generic.go:356] "Generic (PLEG): container finished" podID="fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3" containerID="0b037ed0b40d940a714d3aa685213b7be86834996a19c71097946f2af9d51df3" exitCode=0 Oct 30 21:42:06.934282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:06.934230 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" event={"ID":"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3","Type":"ContainerDied","Data":"0b037ed0b40d940a714d3aa685213b7be86834996a19c71097946f2af9d51df3"} Oct 30 21:42:07.596211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.596176 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz"] Oct 30 21:42:07.600924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.600905 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.651659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.651452 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bld\" (UniqueName: \"kubernetes.io/projected/a1cfe939-1537-4af8-b67a-c9d701566b33-kube-api-access-85bld\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.651849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.651735 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.651955 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.651929 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.658928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.658899 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz"] Oct 30 21:42:07.753028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.752976 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.753239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.753103 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.753239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.753153 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bld\" (UniqueName: \"kubernetes.io/projected/a1cfe939-1537-4af8-b67a-c9d701566b33-kube-api-access-85bld\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.753804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.753783 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.755740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.755719 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.810089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.810058 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bld\" (UniqueName: \"kubernetes.io/projected/a1cfe939-1537-4af8-b67a-c9d701566b33-kube-api-access-85bld\") pod \"maas-api-revoke-sub-39d81bd7-6dxfz\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:07.915696 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:07.915612 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:08.159834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.159807 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:08.259481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.259398 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-openshift-ca\") pod \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " Oct 30 21:42:08.259481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.259463 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-xks-ca\") pod \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " Oct 30 21:42:08.259700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.259572 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-kube-api-access-f8q2m\" (UniqueName: \"kubernetes.io/projected/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-kube-api-access-f8q2m\") pod \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\" (UID: \"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3\") " Oct 30 21:42:08.259862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.259836 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-openshift-ca" pod "fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3" (UID: "fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:08.260218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.260154 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:08.261796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.261764 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-xks-ca" pod "fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3" (UID: "fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:08.261886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.261828 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-kube-api-access-f8q2m" pod "fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3" (UID: "fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3"). InnerVolumeSpecName "kube-api-access-f8q2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:08.314244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.313905 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz"] Oct 30 21:42:08.361863 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.361839 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:08.361863 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.361863 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8q2m\" (UniqueName: \"kubernetes.io/projected/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3-kube-api-access-f8q2m\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:08.952577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.952532 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" event={"ID":"a1cfe939-1537-4af8-b67a-c9d701566b33","Type":"ContainerStarted","Data":"efe47ec719b3c6a0175829c3cd49d7bad5fd5b361d96e45892e54d8e75c7cb52"} Oct 30 21:42:08.952801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.952596 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" event={"ID":"a1cfe939-1537-4af8-b67a-c9d701566b33","Type":"ContainerStarted","Data":"b5c3768128b7abe773030088abe40a9c35cb33b8d5a9e4c69c9ddd09a377bb2c"} Oct 30 21:42:08.954068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.954048 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" event={"ID":"fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3","Type":"ContainerDied","Data":"e1376d53de871fcbba60a8d1b3cc79cfc2ba2904e05fad0ef8e7879718d5d32c"} Oct 30 21:42:08.954160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.954072 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb" Oct 30 21:42:08.954160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:08.954086 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1376d53de871fcbba60a8d1b3cc79cfc2ba2904e05fad0ef8e7879718d5d32c" Oct 30 21:42:09.071063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:09.071011 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" podStartSLOduration=2.070996287 podStartE2EDuration="2.070996287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:09.065660589 +0000 UTC m=+1519.199033788" watchObservedRunningTime="2026-10-30 21:42:09.070996287 +0000 UTC m=+1519.204369510" Oct 30 21:42:09.963974 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:09.963936 4526 generic.go:356] "Generic (PLEG): container finished" podID="a1cfe939-1537-4af8-b67a-c9d701566b33" containerID="efe47ec719b3c6a0175829c3cd49d7bad5fd5b361d96e45892e54d8e75c7cb52" exitCode=0 Oct 30 21:42:09.964399 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:09.964028 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" event={"ID":"a1cfe939-1537-4af8-b67a-c9d701566b33","Type":"ContainerDied","Data":"efe47ec719b3c6a0175829c3cd49d7bad5fd5b361d96e45892e54d8e75c7cb52"} Oct 30 21:42:11.145880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.145857 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:11.207073 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.207041 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1cfe939-1537-4af8-b67a-c9d701566b33-kube-api-access-85bld\" (UniqueName: \"kubernetes.io/projected/a1cfe939-1537-4af8-b67a-c9d701566b33-kube-api-access-85bld\") pod \"a1cfe939-1537-4af8-b67a-c9d701566b33\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " Oct 30 21:42:11.207281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.207138 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-openshift-ca\") pod \"a1cfe939-1537-4af8-b67a-c9d701566b33\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " Oct 30 21:42:11.207281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.207191 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-xks-ca\") pod \"a1cfe939-1537-4af8-b67a-c9d701566b33\" (UID: \"a1cfe939-1537-4af8-b67a-c9d701566b33\") " Oct 30 21:42:11.207576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.207550 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-openshift-ca" pod "a1cfe939-1537-4af8-b67a-c9d701566b33" (UID: "a1cfe939-1537-4af8-b67a-c9d701566b33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:11.208018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.207998 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:11.209437 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.209414 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1cfe939-1537-4af8-b67a-c9d701566b33-kube-api-access-85bld" pod "a1cfe939-1537-4af8-b67a-c9d701566b33" (UID: "a1cfe939-1537-4af8-b67a-c9d701566b33"). InnerVolumeSpecName "kube-api-access-85bld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:11.209563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.209454 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-xks-ca" pod "a1cfe939-1537-4af8-b67a-c9d701566b33" (UID: "a1cfe939-1537-4af8-b67a-c9d701566b33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:11.309923 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.309879 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1cfe939-1537-4af8-b67a-c9d701566b33-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:11.309923 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.309912 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85bld\" (UniqueName: \"kubernetes.io/projected/a1cfe939-1537-4af8-b67a-c9d701566b33-kube-api-access-85bld\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:11.531672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.531636 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9"] Oct 30 21:42:11.532362 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.532333 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a1cfe939-1537-4af8-b67a-c9d701566b33" containerName="revoke-keys" Oct 30 21:42:11.532362 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.532352 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1cfe939-1537-4af8-b67a-c9d701566b33" containerName="revoke-keys" Oct 30 21:42:11.532597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.532386 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3" containerName="revoke-keys" Oct 30 21:42:11.532597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.532393 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3" containerName="revoke-keys" Oct 30 21:42:11.532719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.532626 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1cfe939-1537-4af8-b67a-c9d701566b33" containerName="revoke-keys" Oct 30 21:42:11.532719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.532647 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3" containerName="revoke-keys" Oct 30 21:42:11.536553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.536533 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.595610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.595512 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9"] Oct 30 21:42:11.613972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.613941 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.614139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.614013 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvgv2\" (UniqueName: \"kubernetes.io/projected/67447f64-129a-4bab-9c0d-0b3837cf296a-kube-api-access-rvgv2\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.614187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.614126 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.715440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.715400 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.715686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.715642 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvgv2\" (UniqueName: \"kubernetes.io/projected/67447f64-129a-4bab-9c0d-0b3837cf296a-kube-api-access-rvgv2\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.716019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.715994 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.716170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.716140 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.718593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.718575 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.773463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.773429 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvgv2\" (UniqueName: \"kubernetes.io/projected/67447f64-129a-4bab-9c0d-0b3837cf296a-kube-api-access-rvgv2\") pod \"maas-api-revoke-keys-e2e-ait-bed6793d-xglv9\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.852516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.852384 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:11.986536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.986351 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" Oct 30 21:42:11.988071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.988040 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz" event={"ID":"a1cfe939-1537-4af8-b67a-c9d701566b33","Type":"ContainerDied","Data":"b5c3768128b7abe773030088abe40a9c35cb33b8d5a9e4c69c9ddd09a377bb2c"} Oct 30 21:42:11.988232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:11.988096 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5c3768128b7abe773030088abe40a9c35cb33b8d5a9e4c69c9ddd09a377bb2c" Oct 30 21:42:12.058822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.058786 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt"] Oct 30 21:42:12.067601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.067568 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b26da4e2-rwtvt"] Oct 30 21:42:12.241534 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:42:12.241476 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67447f64_129a_4bab_9c0d_0b3837cf296a.slice/crio-17f90a4800c609493f252ce63e3f24065eb2144d23813898368339777414123e WatchSource:0}: Error finding container 17f90a4800c609493f252ce63e3f24065eb2144d23813898368339777414123e: Status 404 returned error can't find the container with id 17f90a4800c609493f252ce63e3f24065eb2144d23813898368339777414123e Oct 30 21:42:12.242177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.242156 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9"] Oct 30 21:42:12.478866 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:42:12.478829 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:42:12.528319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.528278 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj"] Oct 30 21:42:12.534508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.534468 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.625427 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.625392 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj"] Oct 30 21:42:12.634003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.630392 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.634003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.630553 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577xk\" (UniqueName: \"kubernetes.io/projected/475fdfe9-1408-451c-96e2-520d2bb36454-kube-api-access-577xk\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.634003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.630657 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.732509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.732452 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.732721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.732578 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577xk\" (UniqueName: \"kubernetes.io/projected/475fdfe9-1408-451c-96e2-520d2bb36454-kube-api-access-577xk\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.732721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.732662 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.733200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.733178 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.735290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.735265 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.793309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.793284 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-577xk\" (UniqueName: \"kubernetes.io/projected/475fdfe9-1408-451c-96e2-520d2bb36454-kube-api-access-577xk\") pod \"maas-api-revoke-keys-e2e-derive-385396af-sc8qj\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:12.850062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:12.850015 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:13.005726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:13.005690 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" event={"ID":"67447f64-129a-4bab-9c0d-0b3837cf296a","Type":"ContainerStarted","Data":"7fc8a63e84c684deddcd4f75670bc95afd17641e50fd9806e1334671faa7fdba"} Oct 30 21:42:13.005904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:13.005744 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" event={"ID":"67447f64-129a-4bab-9c0d-0b3837cf296a","Type":"ContainerStarted","Data":"17f90a4800c609493f252ce63e3f24065eb2144d23813898368339777414123e"} Oct 30 21:42:13.126330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:13.126209 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" podStartSLOduration=2.126191934 podStartE2EDuration="2.126191934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:13.119393826 +0000 UTC m=+1523.252767026" watchObservedRunningTime="2026-10-30 21:42:13.126191934 +0000 UTC m=+1523.259565132" Oct 30 21:42:13.239460 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:42:13.239424 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475fdfe9_1408_451c_96e2_520d2bb36454.slice/crio-3b064035a0db8dc1f804dad32cce767cd20f998db25967d23f7ea23a84ea6791 WatchSource:0}: Error finding container 3b064035a0db8dc1f804dad32cce767cd20f998db25967d23f7ea23a84ea6791: Status 404 returned error can't find the container with id 3b064035a0db8dc1f804dad32cce767cd20f998db25967d23f7ea23a84ea6791 Oct 30 21:42:13.241309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:13.241236 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj"] Oct 30 21:42:13.987397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:13.987365 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db135340-6f8f-4de0-adfa-a463ef6e6db4" path="/var/lib/kubelet/pods/db135340-6f8f-4de0-adfa-a463ef6e6db4/volumes" Oct 30 21:42:14.015302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:14.015269 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" event={"ID":"475fdfe9-1408-451c-96e2-520d2bb36454","Type":"ContainerStarted","Data":"8b8a367c40c33dea0481562e9f883f978310273e241f03b97e526e2b3fc28c70"} Oct 30 21:42:14.015469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:14.015325 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" event={"ID":"475fdfe9-1408-451c-96e2-520d2bb36454","Type":"ContainerStarted","Data":"3b064035a0db8dc1f804dad32cce767cd20f998db25967d23f7ea23a84ea6791"} Oct 30 21:42:14.017120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:14.017092 4526 generic.go:356] "Generic (PLEG): container finished" podID="67447f64-129a-4bab-9c0d-0b3837cf296a" containerID="7fc8a63e84c684deddcd4f75670bc95afd17641e50fd9806e1334671faa7fdba" exitCode=0 Oct 30 21:42:14.017244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:14.017172 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" event={"ID":"67447f64-129a-4bab-9c0d-0b3837cf296a","Type":"ContainerDied","Data":"7fc8a63e84c684deddcd4f75670bc95afd17641e50fd9806e1334671faa7fdba"} Oct 30 21:42:14.133364 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:14.133316 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" podStartSLOduration=2.133301098 podStartE2EDuration="2.133301098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:14.127998391 +0000 UTC m=+1524.261371588" watchObservedRunningTime="2026-10-30 21:42:14.133301098 +0000 UTC m=+1524.266674295" Oct 30 21:42:15.027401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.027364 4526 generic.go:356] "Generic (PLEG): container finished" podID="475fdfe9-1408-451c-96e2-520d2bb36454" containerID="8b8a367c40c33dea0481562e9f883f978310273e241f03b97e526e2b3fc28c70" exitCode=0 Oct 30 21:42:15.027880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.027446 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" event={"ID":"475fdfe9-1408-451c-96e2-520d2bb36454","Type":"ContainerDied","Data":"8b8a367c40c33dea0481562e9f883f978310273e241f03b97e526e2b3fc28c70"} Oct 30 21:42:15.201924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.201894 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:15.272126 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.272090 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-openshift-ca\") pod \"67447f64-129a-4bab-9c0d-0b3837cf296a\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " Oct 30 21:42:15.272310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.272228 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-xks-ca\") pod \"67447f64-129a-4bab-9c0d-0b3837cf296a\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " Oct 30 21:42:15.272352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.272306 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67447f64-129a-4bab-9c0d-0b3837cf296a-kube-api-access-rvgv2\" (UniqueName: \"kubernetes.io/projected/67447f64-129a-4bab-9c0d-0b3837cf296a-kube-api-access-rvgv2\") pod \"67447f64-129a-4bab-9c0d-0b3837cf296a\" (UID: \"67447f64-129a-4bab-9c0d-0b3837cf296a\") " Oct 30 21:42:15.272575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.272548 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-openshift-ca" pod "67447f64-129a-4bab-9c0d-0b3837cf296a" (UID: "67447f64-129a-4bab-9c0d-0b3837cf296a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:15.272990 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.272971 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:15.274541 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.274507 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-xks-ca" pod "67447f64-129a-4bab-9c0d-0b3837cf296a" (UID: "67447f64-129a-4bab-9c0d-0b3837cf296a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:15.274632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.274543 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67447f64-129a-4bab-9c0d-0b3837cf296a-kube-api-access-rvgv2" pod "67447f64-129a-4bab-9c0d-0b3837cf296a" (UID: "67447f64-129a-4bab-9c0d-0b3837cf296a"). InnerVolumeSpecName "kube-api-access-rvgv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:15.374092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.374061 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67447f64-129a-4bab-9c0d-0b3837cf296a-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:15.374092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:15.374092 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvgv2\" (UniqueName: \"kubernetes.io/projected/67447f64-129a-4bab-9c0d-0b3837cf296a-kube-api-access-rvgv2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:16.038733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.038701 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" Oct 30 21:42:16.039198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.038744 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9" event={"ID":"67447f64-129a-4bab-9c0d-0b3837cf296a","Type":"ContainerDied","Data":"17f90a4800c609493f252ce63e3f24065eb2144d23813898368339777414123e"} Oct 30 21:42:16.039198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.038790 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17f90a4800c609493f252ce63e3f24065eb2144d23813898368339777414123e" Oct 30 21:42:16.224123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.224100 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:16.286188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.286161 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475fdfe9-1408-451c-96e2-520d2bb36454-kube-api-access-577xk\" (UniqueName: \"kubernetes.io/projected/475fdfe9-1408-451c-96e2-520d2bb36454-kube-api-access-577xk\") pod \"475fdfe9-1408-451c-96e2-520d2bb36454\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " Oct 30 21:42:16.286365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.286286 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-openshift-ca\") pod \"475fdfe9-1408-451c-96e2-520d2bb36454\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " Oct 30 21:42:16.286365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.286315 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-xks-ca\") pod \"475fdfe9-1408-451c-96e2-520d2bb36454\" (UID: \"475fdfe9-1408-451c-96e2-520d2bb36454\") " Oct 30 21:42:16.286729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.286700 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-openshift-ca" pod "475fdfe9-1408-451c-96e2-520d2bb36454" (UID: "475fdfe9-1408-451c-96e2-520d2bb36454"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:16.287029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.287010 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:16.288597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.288574 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-xks-ca" pod "475fdfe9-1408-451c-96e2-520d2bb36454" (UID: "475fdfe9-1408-451c-96e2-520d2bb36454"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:16.288597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.288585 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475fdfe9-1408-451c-96e2-520d2bb36454-kube-api-access-577xk" pod "475fdfe9-1408-451c-96e2-520d2bb36454" (UID: "475fdfe9-1408-451c-96e2-520d2bb36454"). InnerVolumeSpecName "kube-api-access-577xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:16.388241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.388197 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-577xk\" (UniqueName: \"kubernetes.io/projected/475fdfe9-1408-451c-96e2-520d2bb36454-kube-api-access-577xk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:16.388241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:16.388229 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475fdfe9-1408-451c-96e2-520d2bb36454-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:17.051915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:17.051885 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" Oct 30 21:42:17.052297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:17.051948 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj" event={"ID":"475fdfe9-1408-451c-96e2-520d2bb36454","Type":"ContainerDied","Data":"3b064035a0db8dc1f804dad32cce767cd20f998db25967d23f7ea23a84ea6791"} Oct 30 21:42:17.052297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:17.051997 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b064035a0db8dc1f804dad32cce767cd20f998db25967d23f7ea23a84ea6791" Oct 30 21:42:23.128733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:23.128689 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm"] Oct 30 21:42:25.601352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:25.601316 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf"] Oct 30 21:42:25.714737 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:25.714704 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t"] Oct 30 21:42:27.050910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:27.050873 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs"] Oct 30 21:42:27.056794 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:27.056768 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22b6a5f-h8qzs"] Oct 30 21:42:27.987572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:27.987536 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d793c46d-16e0-4229-896f-0cf382f07deb" path="/var/lib/kubelet/pods/d793c46d-16e0-4229-896f-0cf382f07deb/volumes" Oct 30 21:42:28.129984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:28.129931 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" podUID="7a65c7c3-4ef5-4041-aa48-d9de167baf8e" containerName="payload-pre-processing" containerID="cri-o://c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8" gracePeriod=25 Oct 30 21:42:28.507693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:28.507615 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:42:28.679011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:28.678978 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-plugins-config-volume\") pod \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\" (UID: \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\") " Oct 30 21:42:28.679371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:28.679337 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-kube-api-access-5t8h7\" (UniqueName: \"kubernetes.io/projected/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-kube-api-access-5t8h7\") pod \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\" (UID: \"7a65c7c3-4ef5-4041-aa48-d9de167baf8e\") " Oct 30 21:42:28.679520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:28.679417 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-plugins-config-volume" pod "7a65c7c3-4ef5-4041-aa48-d9de167baf8e" (UID: "7a65c7c3-4ef5-4041-aa48-d9de167baf8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:28.680162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:28.680140 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:28.683727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:28.683696 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-kube-api-access-5t8h7" pod "7a65c7c3-4ef5-4041-aa48-d9de167baf8e" (UID: "7a65c7c3-4ef5-4041-aa48-d9de167baf8e"). InnerVolumeSpecName "kube-api-access-5t8h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:28.782208 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:28.782129 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8h7\" (UniqueName: \"kubernetes.io/projected/7a65c7c3-4ef5-4041-aa48-d9de167baf8e-kube-api-access-5t8h7\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:29.242625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.242589 4526 generic.go:356] "Generic (PLEG): container finished" podID="7a65c7c3-4ef5-4041-aa48-d9de167baf8e" containerID="c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8" exitCode=0 Oct 30 21:42:29.243211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.242649 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" Oct 30 21:42:29.243211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.242681 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" event={"ID":"7a65c7c3-4ef5-4041-aa48-d9de167baf8e","Type":"ContainerDied","Data":"c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8"} Oct 30 21:42:29.243211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.242737 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm" event={"ID":"7a65c7c3-4ef5-4041-aa48-d9de167baf8e","Type":"ContainerDied","Data":"19fb0ac11edaef8e75d91f0bbd784d0e76d22f5eb584a2f6db74408f1ce15191"} Oct 30 21:42:29.243211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.242763 4526 scope.go:122] "RemoveContainer" containerID="c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8" Oct 30 21:42:29.254765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.254747 4526 scope.go:122] "RemoveContainer" containerID="c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8" Oct 30 21:42:29.255053 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:42:29.255033 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8\": container with ID starting with c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8 not found: ID does not exist" containerID="c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8" Oct 30 21:42:29.255120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.255062 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8"} err="failed to get container status \"c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8\": rpc error: code = NotFound desc = could not find container \"c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8\": container with ID starting with c7fe2a82096027d3c697c744b5722082e8eff9c03989341e0fd0f445948822f8 not found: ID does not exist" Oct 30 21:42:29.415237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.415207 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm"] Oct 30 21:42:29.464579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.464543 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-385396af-7b485d4b5f-9qcpm"] Oct 30 21:42:29.986982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:29.986952 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a65c7c3-4ef5-4041-aa48-d9de167baf8e" path="/var/lib/kubelet/pods/7a65c7c3-4ef5-4041-aa48-d9de167baf8e/volumes" Oct 30 21:42:30.602674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:30.601773 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" podUID="b9116af5-a82b-4c24-90f1-4f88ce419297" containerName="payload-processing" containerID="cri-o://1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748" gracePeriod=25 Oct 30 21:42:30.715667 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:30.715605 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" podUID="97131b48-6654-45a4-a1f1-decdc0c60b58" containerName="payload-pre-processing" containerID="cri-o://faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd" gracePeriod=25 Oct 30 21:42:31.035476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.035449 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:42:31.080290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.080266 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:42:31.120458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.120428 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97131b48-6654-45a4-a1f1-decdc0c60b58-kube-api-access-fpkxg\" (UniqueName: \"kubernetes.io/projected/97131b48-6654-45a4-a1f1-decdc0c60b58-kube-api-access-fpkxg\") pod \"97131b48-6654-45a4-a1f1-decdc0c60b58\" (UID: \"97131b48-6654-45a4-a1f1-decdc0c60b58\") " Oct 30 21:42:31.120670 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.120477 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97131b48-6654-45a4-a1f1-decdc0c60b58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97131b48-6654-45a4-a1f1-decdc0c60b58-plugins-config-volume\") pod \"97131b48-6654-45a4-a1f1-decdc0c60b58\" (UID: \"97131b48-6654-45a4-a1f1-decdc0c60b58\") " Oct 30 21:42:31.120721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.120697 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9116af5-a82b-4c24-90f1-4f88ce419297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9116af5-a82b-4c24-90f1-4f88ce419297-plugins-config-volume\") pod \"b9116af5-a82b-4c24-90f1-4f88ce419297\" (UID: \"b9116af5-a82b-4c24-90f1-4f88ce419297\") " Oct 30 21:42:31.120875 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.120857 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9116af5-a82b-4c24-90f1-4f88ce419297-kube-api-access-jsgnk\" (UniqueName: \"kubernetes.io/projected/b9116af5-a82b-4c24-90f1-4f88ce419297-kube-api-access-jsgnk\") pod \"b9116af5-a82b-4c24-90f1-4f88ce419297\" (UID: \"b9116af5-a82b-4c24-90f1-4f88ce419297\") " Oct 30 21:42:31.120931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.120899 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97131b48-6654-45a4-a1f1-decdc0c60b58-plugins-config-volume" pod "97131b48-6654-45a4-a1f1-decdc0c60b58" (UID: "97131b48-6654-45a4-a1f1-decdc0c60b58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:31.121035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.121012 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9116af5-a82b-4c24-90f1-4f88ce419297-plugins-config-volume" pod "b9116af5-a82b-4c24-90f1-4f88ce419297" (UID: "b9116af5-a82b-4c24-90f1-4f88ce419297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:31.121436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.121419 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9116af5-a82b-4c24-90f1-4f88ce419297-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:31.121544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.121440 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97131b48-6654-45a4-a1f1-decdc0c60b58-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:31.122837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.122815 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97131b48-6654-45a4-a1f1-decdc0c60b58-kube-api-access-fpkxg" pod "97131b48-6654-45a4-a1f1-decdc0c60b58" (UID: "97131b48-6654-45a4-a1f1-decdc0c60b58"). InnerVolumeSpecName "kube-api-access-fpkxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:31.123125 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.123104 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9116af5-a82b-4c24-90f1-4f88ce419297-kube-api-access-jsgnk" pod "b9116af5-a82b-4c24-90f1-4f88ce419297" (UID: "b9116af5-a82b-4c24-90f1-4f88ce419297"). InnerVolumeSpecName "kube-api-access-jsgnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:31.223141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.223061 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkxg\" (UniqueName: \"kubernetes.io/projected/97131b48-6654-45a4-a1f1-decdc0c60b58-kube-api-access-fpkxg\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:31.223141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.223089 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsgnk\" (UniqueName: \"kubernetes.io/projected/b9116af5-a82b-4c24-90f1-4f88ce419297-kube-api-access-jsgnk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:31.264614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.264572 4526 generic.go:356] "Generic (PLEG): container finished" podID="b9116af5-a82b-4c24-90f1-4f88ce419297" containerID="1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748" exitCode=0 Oct 30 21:42:31.264826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.264634 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" Oct 30 21:42:31.264826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.264661 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" event={"ID":"b9116af5-a82b-4c24-90f1-4f88ce419297","Type":"ContainerDied","Data":"1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748"} Oct 30 21:42:31.264826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.264755 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf" event={"ID":"b9116af5-a82b-4c24-90f1-4f88ce419297","Type":"ContainerDied","Data":"b68f4c7c5cab67e1980d5f3cf9b13b7881e8b87a00b272452990453c8ed3c22e"} Oct 30 21:42:31.264826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.264786 4526 scope.go:122] "RemoveContainer" containerID="1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748" Oct 30 21:42:31.268524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.267777 4526 generic.go:356] "Generic (PLEG): container finished" podID="97131b48-6654-45a4-a1f1-decdc0c60b58" containerID="faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd" exitCode=0 Oct 30 21:42:31.268524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.267834 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" event={"ID":"97131b48-6654-45a4-a1f1-decdc0c60b58","Type":"ContainerDied","Data":"faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd"} Oct 30 21:42:31.268524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.267876 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" event={"ID":"97131b48-6654-45a4-a1f1-decdc0c60b58","Type":"ContainerDied","Data":"f0dbf821f8bf20842b25a222d265cd848afe6b2477cc26a02905641d560d1c22"} Oct 30 21:42:31.268524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.267943 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t" Oct 30 21:42:31.280445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.280411 4526 scope.go:122] "RemoveContainer" containerID="1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748" Oct 30 21:42:31.280888 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:42:31.280859 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748\": container with ID starting with 1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748 not found: ID does not exist" containerID="1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748" Oct 30 21:42:31.281000 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.280899 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748"} err="failed to get container status \"1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748\": rpc error: code = NotFound desc = could not find container \"1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748\": container with ID starting with 1bbed41e04f39679467cb328570dbf69f80366571e416248c4d0f70071cdc748 not found: ID does not exist" Oct 30 21:42:31.281000 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.280925 4526 scope.go:122] "RemoveContainer" containerID="faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd" Oct 30 21:42:31.296075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.296050 4526 scope.go:122] "RemoveContainer" containerID="faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd" Oct 30 21:42:31.296480 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:42:31.296458 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd\": container with ID starting with faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd not found: ID does not exist" containerID="faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd" Oct 30 21:42:31.296617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.296506 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd"} err="failed to get container status \"faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd\": rpc error: code = NotFound desc = could not find container \"faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd\": container with ID starting with faf0f03d3f9b3cffc2d14aa55df04d1ff2c3e27b09be5d43abd389e74ea598dd not found: ID does not exist" Oct 30 21:42:31.434115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.434085 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf"] Oct 30 21:42:31.439302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.439278 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bed6793d-55c96cb599-fxkmf"] Oct 30 21:42:31.548120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.548091 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t"] Oct 30 21:42:31.556367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.556339 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bed6793d-5d84c8c5bc-qxh2t"] Oct 30 21:42:31.987051 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.986958 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97131b48-6654-45a4-a1f1-decdc0c60b58" path="/var/lib/kubelet/pods/97131b48-6654-45a4-a1f1-decdc0c60b58/volumes" Oct 30 21:42:31.987730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:31.987703 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9116af5-a82b-4c24-90f1-4f88ce419297" path="/var/lib/kubelet/pods/b9116af5-a82b-4c24-90f1-4f88ce419297/volumes" Oct 30 21:42:37.063890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.063858 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6"] Oct 30 21:42:37.064700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064672 4526 cpu_manager.go:414] "containerMap: removing container" podUID="475fdfe9-1408-451c-96e2-520d2bb36454" containerName="revoke-keys" Oct 30 21:42:37.064700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064691 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475fdfe9-1408-451c-96e2-520d2bb36454" containerName="revoke-keys" Oct 30 21:42:37.064869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064706 4526 cpu_manager.go:414] "containerMap: removing container" podUID="97131b48-6654-45a4-a1f1-decdc0c60b58" containerName="payload-pre-processing" Oct 30 21:42:37.064869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064711 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97131b48-6654-45a4-a1f1-decdc0c60b58" containerName="payload-pre-processing" Oct 30 21:42:37.064869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064740 4526 cpu_manager.go:414] "containerMap: removing container" podUID="b9116af5-a82b-4c24-90f1-4f88ce419297" containerName="payload-processing" Oct 30 21:42:37.064869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064747 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9116af5-a82b-4c24-90f1-4f88ce419297" containerName="payload-processing" Oct 30 21:42:37.064869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064757 4526 cpu_manager.go:414] "containerMap: removing container" podUID="67447f64-129a-4bab-9c0d-0b3837cf296a" containerName="revoke-keys" Oct 30 21:42:37.064869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064762 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67447f64-129a-4bab-9c0d-0b3837cf296a" containerName="revoke-keys" Oct 30 21:42:37.064869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064782 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7a65c7c3-4ef5-4041-aa48-d9de167baf8e" containerName="payload-pre-processing" Oct 30 21:42:37.064869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064789 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a65c7c3-4ef5-4041-aa48-d9de167baf8e" containerName="payload-pre-processing" Oct 30 21:42:37.065145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064940 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="67447f64-129a-4bab-9c0d-0b3837cf296a" containerName="revoke-keys" Oct 30 21:42:37.065145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064951 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a65c7c3-4ef5-4041-aa48-d9de167baf8e" containerName="payload-pre-processing" Oct 30 21:42:37.065145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064964 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="475fdfe9-1408-451c-96e2-520d2bb36454" containerName="revoke-keys" Oct 30 21:42:37.065145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064977 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="97131b48-6654-45a4-a1f1-decdc0c60b58" containerName="payload-pre-processing" Oct 30 21:42:37.065145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.064990 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9116af5-a82b-4c24-90f1-4f88ce419297" containerName="payload-processing" Oct 30 21:42:37.070608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.070576 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:37.122134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.122100 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcaf4b-8e8e-458a-bac1-41bf91806289-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6\" (UID: \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:37.122338 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.122189 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m484\" (UniqueName: \"kubernetes.io/projected/d9fcaf4b-8e8e-458a-bac1-41bf91806289-kube-api-access-4m484\") pod \"payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6\" (UID: \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:37.125229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.125202 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6"] Oct 30 21:42:37.166843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.166810 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-3b21b4\"" Oct 30 21:42:37.167061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.166902 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-3b21b4-dockercfg-t48mr\"" Oct 30 21:42:37.222589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.222560 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx"] Oct 30 21:42:37.223471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.223448 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcaf4b-8e8e-458a-bac1-41bf91806289-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6\" (UID: \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:37.223616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.223509 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m484\" (UniqueName: \"kubernetes.io/projected/d9fcaf4b-8e8e-458a-bac1-41bf91806289-kube-api-access-4m484\") pod \"payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6\" (UID: \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:37.224164 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.224136 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcaf4b-8e8e-458a-bac1-41bf91806289-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6\" (UID: \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:37.227370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.227345 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:37.281695 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.281663 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m484\" (UniqueName: \"kubernetes.io/projected/d9fcaf4b-8e8e-458a-bac1-41bf91806289-kube-api-access-4m484\") pod \"payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6\" (UID: \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:37.287101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.287069 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx"] Oct 30 21:42:37.331517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.325194 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx\" (UID: \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:37.331517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.325283 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzkb\" (UniqueName: \"kubernetes.io/projected/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-kube-api-access-6qzkb\") pod \"payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx\" (UID: \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:37.395300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.395258 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:37.427772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.427740 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx\" (UID: \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:37.427934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.427801 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzkb\" (UniqueName: \"kubernetes.io/projected/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-kube-api-access-6qzkb\") pod \"payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx\" (UID: \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:37.428712 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.428690 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx\" (UID: \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:37.498526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.491219 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzkb\" (UniqueName: \"kubernetes.io/projected/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-kube-api-access-6qzkb\") pod \"payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx\" (UID: \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:37.568575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.555449 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:37.994883 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:42:37.994852 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fcaf4b_8e8e_458a_bac1_41bf91806289.slice/crio-768f5c620d94c09104ccd22e3cc99254caf83b35c3702da36b16e94536a30765 WatchSource:0}: Error finding container 768f5c620d94c09104ccd22e3cc99254caf83b35c3702da36b16e94536a30765: Status 404 returned error can't find the container with id 768f5c620d94c09104ccd22e3cc99254caf83b35c3702da36b16e94536a30765 Oct 30 21:42:37.997536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:37.997482 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6"] Oct 30 21:42:38.118558 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:42:38.118482 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4439a8_9ca4_4f02_862e_7ec85e0f9561.slice/crio-0b73efee691d6351ebd0052b297e7e92a402eaf81165a721b03a622039d04306 WatchSource:0}: Error finding container 0b73efee691d6351ebd0052b297e7e92a402eaf81165a721b03a622039d04306: Status 404 returned error can't find the container with id 0b73efee691d6351ebd0052b297e7e92a402eaf81165a721b03a622039d04306 Oct 30 21:42:38.119054 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.118979 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx"] Oct 30 21:42:38.343590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.343550 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" event={"ID":"5b4439a8-9ca4-4f02-862e-7ec85e0f9561","Type":"ContainerStarted","Data":"939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a"} Oct 30 21:42:38.343821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.343617 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" event={"ID":"5b4439a8-9ca4-4f02-862e-7ec85e0f9561","Type":"ContainerStarted","Data":"0b73efee691d6351ebd0052b297e7e92a402eaf81165a721b03a622039d04306"} Oct 30 21:42:38.343821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.343808 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:38.345571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.345542 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" event={"ID":"d9fcaf4b-8e8e-458a-bac1-41bf91806289","Type":"ContainerStarted","Data":"b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df"} Oct 30 21:42:38.346011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.345596 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:38.346011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.345619 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" event={"ID":"d9fcaf4b-8e8e-458a-bac1-41bf91806289","Type":"ContainerStarted","Data":"768f5c620d94c09104ccd22e3cc99254caf83b35c3702da36b16e94536a30765"} Oct 30 21:42:38.460649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.460586 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" podStartSLOduration=1.460571778 podStartE2EDuration="1.460571778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:38.455694443 +0000 UTC m=+1548.589067643" watchObservedRunningTime="2026-10-30 21:42:38.460571778 +0000 UTC m=+1548.593944976" Oct 30 21:42:38.571521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:38.571440 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" podStartSLOduration=1.571417554 podStartE2EDuration="1.571417554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:38.569052588 +0000 UTC m=+1548.702425799" watchObservedRunningTime="2026-10-30 21:42:38.571417554 +0000 UTC m=+1548.704790754" Oct 30 21:42:42.480164 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:42:42.480126 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:42:49.075919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:49.075881 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9"] Oct 30 21:42:49.112821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:49.112780 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bed6793d-xglv9"] Oct 30 21:42:49.118702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:49.118668 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm"] Oct 30 21:42:49.118970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:49.118947 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" containerID="cri-o://dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682" gracePeriod=30 Oct 30 21:42:49.378952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:49.378847 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:42:49.379801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:49.379775 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:42:49.999983 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:49.999952 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67447f64-129a-4bab-9c0d-0b3837cf296a" path="/var/lib/kubelet/pods/67447f64-129a-4bab-9c0d-0b3837cf296a/volumes" Oct 30 21:42:50.497441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:50.497407 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb"] Oct 30 21:42:50.509759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:50.509723 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d56d1199-7brxb"] Oct 30 21:42:51.843078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:51.840969 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj"] Oct 30 21:42:51.903700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:51.902303 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-385396af-sc8qj"] Oct 30 21:42:51.903700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:51.902374 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n"] Oct 30 21:42:51.903700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:51.902656 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" containerID="cri-o://e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23" gracePeriod=30 Oct 30 21:42:51.995712 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:51.995650 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475fdfe9-1408-451c-96e2-520d2bb36454" path="/var/lib/kubelet/pods/475fdfe9-1408-451c-96e2-520d2bb36454/volumes" Oct 30 21:42:51.996472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:51.996442 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3" path="/var/lib/kubelet/pods/fc3ac6b8-d3ec-48e6-90e2-ac87e6527fe3/volumes" Oct 30 21:42:54.448902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.448875 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:42:54.511663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.511623 4526 generic.go:356] "Generic (PLEG): container finished" podID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerID="dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682" exitCode=0 Oct 30 21:42:54.511850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.511671 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" Oct 30 21:42:54.511850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.511721 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" event={"ID":"fff3077f-ff81-4f97-b62a-27618625a9b2","Type":"ContainerDied","Data":"dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682"} Oct 30 21:42:54.511850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.511777 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm" event={"ID":"fff3077f-ff81-4f97-b62a-27618625a9b2","Type":"ContainerDied","Data":"e314e8e88fdda24ae440c26bea9cca4b9a153b500f116a214ece86de2c4d788e"} Oct 30 21:42:54.511850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.511805 4526 scope.go:122] "RemoveContainer" containerID="dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682" Oct 30 21:42:54.522889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.522865 4526 scope.go:122] "RemoveContainer" containerID="dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682" Oct 30 21:42:54.523200 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:42:54.523176 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682\": container with ID starting with dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682 not found: ID does not exist" containerID="dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682" Oct 30 21:42:54.523251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.523213 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682"} err="failed to get container status \"dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682\": rpc error: code = NotFound desc = could not find container \"dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682\": container with ID starting with dba48bd8245ceb8c076e6d38f04d98ff50a08eea88383559ff3cad7163952682 not found: ID does not exist" Oct 30 21:42:54.551042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.550997 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-data\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551063 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-token\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-token\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551121 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff3077f-ff81-4f97-b62a-27618625a9b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff3077f-ff81-4f97-b62a-27618625a9b2-istiod-ca-cert\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551190 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-certs\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551262 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-envoy\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551296 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-data" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.551335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551327 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-podinfo\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551463 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-credential-socket\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551542 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-certs" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.551635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551580 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-kube-api-access-6m6pw\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-kube-api-access-6m6pw\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551609 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff3077f-ff81-4f97-b62a-27618625a9b2-istiod-ca-cert" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:54.551749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551681 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-socket\") pod \"fff3077f-ff81-4f97-b62a-27618625a9b2\" (UID: \"fff3077f-ff81-4f97-b62a-27618625a9b2\") " Oct 30 21:42:54.551835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.551813 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-credential-socket" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.552154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.552130 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-socket" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.552481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.552460 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.552591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.552524 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.552591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.552542 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.552591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.552556 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff3077f-ff81-4f97-b62a-27618625a9b2-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.552591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.552569 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.553959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.553929 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-token" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:54.554066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.553962 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-kube-api-access-6m6pw" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "kube-api-access-6m6pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:54.554121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.554066 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-envoy" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.554157 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.554125 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-podinfo" pod "fff3077f-ff81-4f97-b62a-27618625a9b2" (UID: "fff3077f-ff81-4f97-b62a-27618625a9b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:54.664882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.664785 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.664882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.664842 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.664882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.664860 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff3077f-ff81-4f97-b62a-27618625a9b2-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.664882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.664873 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6pw\" (UniqueName: \"kubernetes.io/projected/fff3077f-ff81-4f97-b62a-27618625a9b2-kube-api-access-6m6pw\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.971586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.971509 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm"] Oct 30 21:42:54.989353 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:54.989311 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-424pm"] Oct 30 21:42:55.987323 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:55.987279 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" path="/var/lib/kubelet/pods/fff3077f-ff81-4f97-b62a-27618625a9b2/volumes" Oct 30 21:42:57.249915 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.249890 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:42:57.312202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312167 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-certs\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.312372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312254 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-kube-api-access-bjf8x\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-kube-api-access-bjf8x\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.312372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312293 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-socket\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.312372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312349 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-istio-token\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-istio-token\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.312480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312441 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-certs" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:57.312480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312457 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-data\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.312620 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312533 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0838b810-8246-4b65-a450-cc33e4cc1121-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838b810-8246-4b65-a450-cc33e4cc1121-istiod-ca-cert\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.312703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312677 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-socket" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:57.312703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312690 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-data" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:57.312819 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312686 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-envoy\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.312884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312820 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-credential-socket\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.312884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312844 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0838b810-8246-4b65-a450-cc33e4cc1121-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838b810-8246-4b65-a450-cc33e4cc1121-istio-podinfo\") pod \"0838b810-8246-4b65-a450-cc33e4cc1121\" (UID: \"0838b810-8246-4b65-a450-cc33e4cc1121\") " Oct 30 21:42:57.313032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.312928 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0838b810-8246-4b65-a450-cc33e4cc1121-istiod-ca-cert" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:57.313093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.313069 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-credential-socket" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:57.313543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.313524 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.313543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.313543 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.313883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.313552 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.313883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.313562 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.313883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.313575 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0838b810-8246-4b65-a450-cc33e4cc1121-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.314869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.314842 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-kube-api-access-bjf8x" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "kube-api-access-bjf8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:57.315217 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.315201 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-istio-token" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:57.315310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.315229 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-envoy" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:57.315423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.315408 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0838b810-8246-4b65-a450-cc33e4cc1121-istio-podinfo" pod "0838b810-8246-4b65-a450-cc33e4cc1121" (UID: "0838b810-8246-4b65-a450-cc33e4cc1121"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:57.414877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.414840 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0838b810-8246-4b65-a450-cc33e4cc1121-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.414877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.414870 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0838b810-8246-4b65-a450-cc33e4cc1121-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.414877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.414881 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjf8x\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-kube-api-access-bjf8x\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.415169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.414890 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0838b810-8246-4b65-a450-cc33e4cc1121-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:57.541115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.541080 4526 generic.go:356] "Generic (PLEG): container finished" podID="0838b810-8246-4b65-a450-cc33e4cc1121" containerID="e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23" exitCode=0 Oct 30 21:42:57.541294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.541132 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" Oct 30 21:42:57.541294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.541164 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" event={"ID":"0838b810-8246-4b65-a450-cc33e4cc1121","Type":"ContainerDied","Data":"e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23"} Oct 30 21:42:57.541294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.541240 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n" event={"ID":"0838b810-8246-4b65-a450-cc33e4cc1121","Type":"ContainerDied","Data":"eaa21b06f8110f47f57c0ffdf3f3c6a7f221b145465f578a24c828296cc85297"} Oct 30 21:42:57.541294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.541265 4526 scope.go:122] "RemoveContainer" containerID="e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23" Oct 30 21:42:57.553330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.553311 4526 scope.go:122] "RemoveContainer" containerID="e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed" Oct 30 21:42:57.563521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.563504 4526 scope.go:122] "RemoveContainer" containerID="e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23" Oct 30 21:42:57.563803 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:42:57.563784 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23\": container with ID starting with e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23 not found: ID does not exist" containerID="e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23" Oct 30 21:42:57.563871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.563815 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23"} err="failed to get container status \"e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23\": rpc error: code = NotFound desc = could not find container \"e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23\": container with ID starting with e709cea59602ac3292f59c069c7a4d49928ca250b036dbb89ea4bda3d66a2c23 not found: ID does not exist" Oct 30 21:42:57.563871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.563834 4526 scope.go:122] "RemoveContainer" containerID="e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed" Oct 30 21:42:57.564086 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:42:57.564070 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed\": container with ID starting with e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed not found: ID does not exist" containerID="e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed" Oct 30 21:42:57.564134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.564091 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed"} err="failed to get container status \"e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed\": rpc error: code = NotFound desc = could not find container \"e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed\": container with ID starting with e2a5906be3b2e98ecee3596d073f273e9a74d715171a5de1aae25a54787561ed not found: ID does not exist" Oct 30 21:42:57.717084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.717047 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n"] Oct 30 21:42:57.722545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.722515 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-385396af-openshift-default-5cf97b5c97-zxc9n"] Oct 30 21:42:57.988005 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:42:57.987973 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" path="/var/lib/kubelet/pods/0838b810-8246-4b65-a450-cc33e4cc1121/volumes" Oct 30 21:43:07.417566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.417464 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh"] Oct 30 21:43:07.418548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418524 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" Oct 30 21:43:07.418548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418547 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" Oct 30 21:43:07.418724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418579 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" Oct 30 21:43:07.418724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418587 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" Oct 30 21:43:07.418724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418605 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" Oct 30 21:43:07.418724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418611 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" Oct 30 21:43:07.418850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418796 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff3077f-ff81-4f97-b62a-27618625a9b2" containerName="istio-proxy" Oct 30 21:43:07.418850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418809 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" Oct 30 21:43:07.418850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.418821 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0838b810-8246-4b65-a450-cc33e4cc1121" containerName="istio-proxy" Oct 30 21:43:07.423921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.423906 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.520987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.520953 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00307618-1d57-4089-aaf6-5b1436741bcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.521179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.521012 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.521179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.521099 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.521274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.521258 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.521332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.521319 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00307618-1d57-4089-aaf6-5b1436741bcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.521460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.521439 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.521527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.521516 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.521593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.521578 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.521646 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.521633 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmw8\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-kube-api-access-hdmw8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623002 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmw8\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-kube-api-access-hdmw8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623120 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00307618-1d57-4089-aaf6-5b1436741bcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623176 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623206 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623357 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623447 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00307618-1d57-4089-aaf6-5b1436741bcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623539 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623577 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623634 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.623843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.623769 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.624042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.624019 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.624616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.624275 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.624616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.624400 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00307618-1d57-4089-aaf6-5b1436741bcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.624616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.624553 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.627871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.627845 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00307618-1d57-4089-aaf6-5b1436741bcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.632042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.629024 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.661728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.661690 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh"] Oct 30 21:43:07.685551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.685432 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmw8\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-kube-api-access-hdmw8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.686976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.686948 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-pg8wh\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:07.738688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:07.738660 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:08.111085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.111048 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv"] Oct 30 21:43:08.123432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.123405 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:08.183272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.183218 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv"] Oct 30 21:43:08.195561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.190883 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1915737c-dockercfg-c8mkv\"" Oct 30 21:43:08.195561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.191589 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1915737c\"" Oct 30 21:43:08.215102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.215065 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:08.215199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.215183 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:08.215255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.215229 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:08.230264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.230240 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh"] Oct 30 21:43:08.250347 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.250321 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbm94\" (UniqueName: \"kubernetes.io/projected/e2793212-6363-4bff-be30-2ce58a0701bb-kube-api-access-rbm94\") pod \"payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv\" (UID: \"e2793212-6363-4bff-be30-2ce58a0701bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:08.250666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.250634 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2793212-6363-4bff-be30-2ce58a0701bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv\" (UID: \"e2793212-6363-4bff-be30-2ce58a0701bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:08.273449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.273418 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz"] Oct 30 21:43:08.284738 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.278510 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:08.360569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.353336 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbm94\" (UniqueName: \"kubernetes.io/projected/e2793212-6363-4bff-be30-2ce58a0701bb-kube-api-access-rbm94\") pod \"payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv\" (UID: \"e2793212-6363-4bff-be30-2ce58a0701bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:08.360569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.353646 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2793212-6363-4bff-be30-2ce58a0701bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv\" (UID: \"e2793212-6363-4bff-be30-2ce58a0701bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:08.360569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.354387 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2793212-6363-4bff-be30-2ce58a0701bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv\" (UID: \"e2793212-6363-4bff-be30-2ce58a0701bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:08.375063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.374987 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz"] Oct 30 21:43:08.456185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.454246 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbm94\" (UniqueName: \"kubernetes.io/projected/e2793212-6363-4bff-be30-2ce58a0701bb-kube-api-access-rbm94\") pod \"payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv\" (UID: \"e2793212-6363-4bff-be30-2ce58a0701bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:08.456185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.455425 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9wl\" (UniqueName: \"kubernetes.io/projected/3c437433-8510-422a-8ac3-ffba19fe30c8-kube-api-access-jq9wl\") pod \"payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz\" (UID: \"3c437433-8510-422a-8ac3-ffba19fe30c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:08.456185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.455646 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c437433-8510-422a-8ac3-ffba19fe30c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz\" (UID: \"3c437433-8510-422a-8ac3-ffba19fe30c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:08.557238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.557205 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9wl\" (UniqueName: \"kubernetes.io/projected/3c437433-8510-422a-8ac3-ffba19fe30c8-kube-api-access-jq9wl\") pod \"payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz\" (UID: \"3c437433-8510-422a-8ac3-ffba19fe30c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:08.557413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.557322 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c437433-8510-422a-8ac3-ffba19fe30c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz\" (UID: \"3c437433-8510-422a-8ac3-ffba19fe30c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:08.558068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.558047 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c437433-8510-422a-8ac3-ffba19fe30c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz\" (UID: \"3c437433-8510-422a-8ac3-ffba19fe30c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:08.615852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.615820 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9wl\" (UniqueName: \"kubernetes.io/projected/3c437433-8510-422a-8ac3-ffba19fe30c8-kube-api-access-jq9wl\") pod \"payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz\" (UID: \"3c437433-8510-422a-8ac3-ffba19fe30c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:08.645638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.645530 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:08.712464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.712291 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" event={"ID":"00307618-1d57-4089-aaf6-5b1436741bcf","Type":"ContainerStarted","Data":"56de2d119a76d3088223439b36b044a77006e86db1d5131e6df909e920fb8dd4"} Oct 30 21:43:08.712464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.712349 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" event={"ID":"00307618-1d57-4089-aaf6-5b1436741bcf","Type":"ContainerStarted","Data":"39c679eda318231d43a80b5552367119f915717f2dd6f9eecd9ab9b3a80b11ce"} Oct 30 21:43:08.739128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.738438 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:08.760610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.760577 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:08.864668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:08.858916 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podStartSLOduration=1.858900213 podStartE2EDuration="1.858900213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:08.840667479 +0000 UTC m=+1578.974040676" watchObservedRunningTime="2026-10-30 21:43:08.858900213 +0000 UTC m=+1578.992273409" Oct 30 21:43:09.042541 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:43:09.042462 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c437433_8510_422a_8ac3_ffba19fe30c8.slice/crio-3297986faef598f23d35c7fee4f4972ef1ec7e1ad5a9231b60b9e030ef35663e WatchSource:0}: Error finding container 3297986faef598f23d35c7fee4f4972ef1ec7e1ad5a9231b60b9e030ef35663e: Status 404 returned error can't find the container with id 3297986faef598f23d35c7fee4f4972ef1ec7e1ad5a9231b60b9e030ef35663e Oct 30 21:43:09.042985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.042958 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz"] Oct 30 21:43:09.178525 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:43:09.176267 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2793212_6363_4bff_be30_2ce58a0701bb.slice/crio-198015cfc1f0adac31e435ad3bd091fd216750b423acdf4b4df0de72a85f39c4 WatchSource:0}: Error finding container 198015cfc1f0adac31e435ad3bd091fd216750b423acdf4b4df0de72a85f39c4: Status 404 returned error can't find the container with id 198015cfc1f0adac31e435ad3bd091fd216750b423acdf4b4df0de72a85f39c4 Oct 30 21:43:09.182843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.182816 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv"] Oct 30 21:43:09.731848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.731807 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" event={"ID":"3c437433-8510-422a-8ac3-ffba19fe30c8","Type":"ContainerStarted","Data":"224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb"} Oct 30 21:43:09.732297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.731873 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" event={"ID":"3c437433-8510-422a-8ac3-ffba19fe30c8","Type":"ContainerStarted","Data":"3297986faef598f23d35c7fee4f4972ef1ec7e1ad5a9231b60b9e030ef35663e"} Oct 30 21:43:09.732297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.731904 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:09.733767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.733740 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" event={"ID":"e2793212-6363-4bff-be30-2ce58a0701bb","Type":"ContainerStarted","Data":"5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21"} Oct 30 21:43:09.733893 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.733801 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" event={"ID":"e2793212-6363-4bff-be30-2ce58a0701bb","Type":"ContainerStarted","Data":"198015cfc1f0adac31e435ad3bd091fd216750b423acdf4b4df0de72a85f39c4"} Oct 30 21:43:09.733893 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.733831 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:09.740932 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.740906 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pg8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 21:43:09.741057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.740961 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 21:43:09.847353 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.847298 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" podStartSLOduration=1.8472829910000002 podStartE2EDuration="1.847282991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:09.846158856 +0000 UTC m=+1579.979532054" watchObservedRunningTime="2026-10-30 21:43:09.847282991 +0000 UTC m=+1579.980656235" Oct 30 21:43:09.972963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:09.972895 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" podStartSLOduration=1.9728745330000002 podStartE2EDuration="1.972874533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:09.96537412 +0000 UTC m=+1580.098747321" watchObservedRunningTime="2026-10-30 21:43:09.972874533 +0000 UTC m=+1580.106247732" Oct 30 21:43:10.057694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:10.057646 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w"] Oct 30 21:43:10.064163 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:10.064134 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b79a13d9-x7r9w"] Oct 30 21:43:10.739745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:10.739703 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pg8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 21:43:10.740190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:10.739766 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 21:43:11.739842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:11.739804 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pg8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 21:43:11.740241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:11.739861 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 21:43:11.987508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:11.987462 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8357554a-60c8-4c16-b4da-331f30a57ee5" path="/var/lib/kubelet/pods/8357554a-60c8-4c16-b4da-331f30a57ee5/volumes" Oct 30 21:43:12.479821 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:43:12.479788 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:43:12.739764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:12.739677 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pg8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 21:43:12.739764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:12.739736 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 21:43:13.739997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:13.739964 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pg8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 21:43:13.740385 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:13.740016 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 21:43:14.739578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:14.739547 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pg8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 21:43:14.739760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:14.739601 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 21:43:15.739949 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:15.739913 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pg8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 21:43:15.740340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:15.739983 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 21:43:17.088645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:17.088615 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:17.089711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:17.089675 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:17.090836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:17.090815 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:43:20.745433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:20.745402 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:43:20.745936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:20.745472 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:43:21.073403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:21.073363 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq"] Oct 30 21:43:21.079850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:21.079820 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93f76d0e-fn6bq"] Oct 30 21:43:21.987254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:21.987218 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f01b59-20f2-4e0f-a3bd-41807c097d88" path="/var/lib/kubelet/pods/13f01b59-20f2-4e0f-a3bd-41807c097d88/volumes" Oct 30 21:43:27.076281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:27.076246 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp"] Oct 30 21:43:27.123432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:27.123389 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f24db36e-mgcdp"] Oct 30 21:43:27.987681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:27.987650 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb00eca-9882-492d-bc92-b26bc53fa01e" path="/var/lib/kubelet/pods/4cb00eca-9882-492d-bc92-b26bc53fa01e/volumes" Oct 30 21:43:32.061759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:32.061723 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl"] Oct 30 21:43:32.111027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:32.110986 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0032967-pw4hl"] Oct 30 21:43:33.724683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:33.724644 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:43:33.725306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:33.725270 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:43:33.987202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:33.987124 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c9ec3b-435e-4e24-ae23-bf4da48820db" path="/var/lib/kubelet/pods/98c9ec3b-435e-4e24-ae23-bf4da48820db/volumes" Oct 30 21:43:42.481790 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:43:42.481754 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:43:46.069662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:46.069624 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh"] Oct 30 21:43:46.119007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:46.118965 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35d721a8-58nsh"] Oct 30 21:43:46.123702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:46.123681 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx"] Oct 30 21:43:46.127972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:46.127945 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7024597b-6d2lx"] Oct 30 21:43:46.928122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:46.928085 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt"] Oct 30 21:43:46.932655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:46.932633 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.030859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.030827 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-2ace1d-openshift-default-dockercfg-9m9qr\"" Oct 30 21:43:47.072515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.063785 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-certs\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.104123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.104094 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-data\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.104476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.104453 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-envoy\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.106609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.104749 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x74p\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-kube-api-access-4x74p\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.106851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.106834 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-credential-socket\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.107045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.106981 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-token\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.107256 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.107216 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-podinfo\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.107587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.107570 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc33b1-e449-4b7f-b1eb-10adff633529-istiod-ca-cert\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.108042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.108009 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-socket\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.118696 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.117431 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt"] Oct 30 21:43:47.210950 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.210771 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc33b1-e449-4b7f-b1eb-10adff633529-istiod-ca-cert\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.210950 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.210873 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-socket\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.211186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.211078 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-certs\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.211186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.211147 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-data\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.211299 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.211201 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-envoy\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.211299 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.211282 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x74p\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-kube-api-access-4x74p\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.217139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.211333 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-credential-socket\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.217139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.211357 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-token\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.217139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.211387 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-podinfo\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.217139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.215277 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-podinfo\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.217139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.216084 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc33b1-e449-4b7f-b1eb-10adff633529-istiod-ca-cert\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.217139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.216356 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-socket\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.217139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.216587 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-certs\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.217139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.216819 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-data\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.218863 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.218838 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-credential-socket\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.220018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.219997 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-envoy\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.277541 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.277502 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x74p\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-kube-api-access-4x74p\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.277700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.277547 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-token\") pod \"e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.551326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.551292 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:47.986601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.986517 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085f232e-e45d-4ddf-abdf-95ad40031bf7" path="/var/lib/kubelet/pods/085f232e-e45d-4ddf-abdf-95ad40031bf7/volumes" Oct 30 21:43:47.986992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:47.986979 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e8e269-2272-48de-999c-fdc531cfbbf7" path="/var/lib/kubelet/pods/f2e8e269-2272-48de-999c-fdc531cfbbf7/volumes" Oct 30 21:43:48.021911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:48.021883 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt"] Oct 30 21:43:48.057166 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:43:48.057140 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dc33b1_e449_4b7f_b1eb_10adff633529.slice/crio-98d5af3b909db08c5bb8b4af06c9123cc62a9129e81feab08eee58fcaa06f9cb WatchSource:0}: Error finding container 98d5af3b909db08c5bb8b4af06c9123cc62a9129e81feab08eee58fcaa06f9cb: Status 404 returned error can't find the container with id 98d5af3b909db08c5bb8b4af06c9123cc62a9129e81feab08eee58fcaa06f9cb Oct 30 21:43:48.059690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:48.059653 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:48.059747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:48.059718 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:48.059783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:48.059751 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:48.116575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:48.116541 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" event={"ID":"f4dc33b1-e449-4b7f-b1eb-10adff633529","Type":"ContainerStarted","Data":"98d5af3b909db08c5bb8b4af06c9123cc62a9129e81feab08eee58fcaa06f9cb"} Oct 30 21:43:49.126447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.126411 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" event={"ID":"f4dc33b1-e449-4b7f-b1eb-10adff633529","Type":"ContainerStarted","Data":"fdb610614e5912c4863a59bc3572df620dda014d9aab572e68590ed7bca497c1"} Oct 30 21:43:49.302187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.302131 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" podStartSLOduration=3.302113828 podStartE2EDuration="3.302113828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:49.295995554 +0000 UTC m=+1619.429368753" watchObservedRunningTime="2026-10-30 21:43:49.302113828 +0000 UTC m=+1619.435487029" Oct 30 21:43:49.502365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.497918 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd"] Oct 30 21:43:49.515825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.515797 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.553734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.553667 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:49.556515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.555297 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd"] Oct 30 21:43:49.564094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.564065 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:49.576618 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.576583 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-29b7ead1-openshift-default-dockercfg-wxt9b\"" Oct 30 21:43:49.577732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.577702 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-certs\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.577882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.577748 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgftq\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-kube-api-access-rgftq\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.577882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.577842 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-token\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.577882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.577870 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-credential-socket\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.578063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.577992 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-socket\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.578123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.578079 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istiod-ca-cert\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.578185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.578158 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-podinfo\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.578244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.578207 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-data\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.578330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.578311 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-envoy\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680062 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-socket\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680151 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istiod-ca-cert\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680234 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-podinfo\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680279 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-data\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680381 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-envoy\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680547 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-certs\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680573 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgftq\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-kube-api-access-rgftq\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680644 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-token\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680672 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-credential-socket\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.680872 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680849 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-certs\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.681006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680981 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istiod-ca-cert\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.681184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.680548 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-socket\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.681184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.681045 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-credential-socket\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.681340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.681311 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-data\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.682926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.682901 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-envoy\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.683480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.683456 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-podinfo\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.746023 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.745991 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-token\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.752911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.746586 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgftq\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-kube-api-access-rgftq\") pod \"e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:49.867908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:49.867874 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:50.137565 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:50.137534 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:50.139578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:50.138962 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:43:50.383866 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:43:50.383833 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc8fdad_213d_4748_94f3_f4c1359c8bbc.slice/crio-cd071a0ca446f83197bac803ca3c3491c46295c8267a00009c076c1563f6c95d WatchSource:0}: Error finding container cd071a0ca446f83197bac803ca3c3491c46295c8267a00009c076c1563f6c95d: Status 404 returned error can't find the container with id cd071a0ca446f83197bac803ca3c3491c46295c8267a00009c076c1563f6c95d Oct 30 21:43:50.384544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:50.384523 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd"] Oct 30 21:43:50.387063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:50.387031 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:50.387154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:50.387099 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:50.387154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:50.387128 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:50.799842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:50.799808 4526 scope.go:122] "RemoveContainer" containerID="abe0230fabbebc2afe6f7c2248a04770e3c8dc28c6b9d7384189106886d33c17" Oct 30 21:43:50.811397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:50.811368 4526 scope.go:122] "RemoveContainer" containerID="debb5ba68ac3f52e100fe054a5e857c473e700c8be10c5b979b9e2633bfc13b1" Oct 30 21:43:51.058929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:51.058828 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6"] Oct 30 21:43:51.064119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:51.064096 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3b100c9-jzzh6"] Oct 30 21:43:51.146836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:51.146795 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" event={"ID":"0cc8fdad-213d-4748-94f3-f4c1359c8bbc","Type":"ContainerStarted","Data":"9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa"} Oct 30 21:43:51.147329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:51.146851 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" event={"ID":"0cc8fdad-213d-4748-94f3-f4c1359c8bbc","Type":"ContainerStarted","Data":"cd071a0ca446f83197bac803ca3c3491c46295c8267a00009c076c1563f6c95d"} Oct 30 21:43:51.275921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:51.275866 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podStartSLOduration=2.275850969 podStartE2EDuration="2.275850969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:51.271612611 +0000 UTC m=+1621.404985832" watchObservedRunningTime="2026-10-30 21:43:51.275850969 +0000 UTC m=+1621.409224222" Oct 30 21:43:51.868966 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:51.868927 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:51.875462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:51.875434 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:51.986937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:51.986903 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d26d0e4-7b64-4f32-8066-57de59d25890" path="/var/lib/kubelet/pods/5d26d0e4-7b64-4f32-8066-57de59d25890/volumes" Oct 30 21:43:52.155648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:52.155550 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:52.156469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:52.156450 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:43:54.080587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:54.080541 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp"] Oct 30 21:43:54.129956 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:54.129918 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff811e63-bwvbp"] Oct 30 21:43:56.002693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.002665 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7d5b5c-7585-4542-b94d-c7770293efb9" path="/var/lib/kubelet/pods/bf7d5b5c-7585-4542-b94d-c7770293efb9/volumes" Oct 30 21:43:56.237224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.237188 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8"] Oct 30 21:43:56.243548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.243528 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:56.320869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.320837 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8"] Oct 30 21:43:56.322429 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.322387 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8\" (UID: \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:56.323046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.322895 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxfw\" (UniqueName: \"kubernetes.io/projected/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-kube-api-access-nfxfw\") pod \"payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8\" (UID: \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:56.390631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.389891 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-2ace1d\"" Oct 30 21:43:56.390631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.390064 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-2ace1d-dockercfg-dzv9s\"" Oct 30 21:43:56.425941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.425899 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8\" (UID: \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:56.426146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.426110 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxfw\" (UniqueName: \"kubernetes.io/projected/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-kube-api-access-nfxfw\") pod \"payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8\" (UID: \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:56.427161 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.427117 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8\" (UID: \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:56.433958 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.433933 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns"] Oct 30 21:43:56.438844 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.438822 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:56.459853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.459823 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns"] Oct 30 21:43:56.485954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.485920 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxfw\" (UniqueName: \"kubernetes.io/projected/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-kube-api-access-nfxfw\") pod \"payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8\" (UID: \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:56.562659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.562351 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:56.650261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.647306 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078760c3-3213-4e7d-9a60-935413e9377a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns\" (UID: \"078760c3-3213-4e7d-9a60-935413e9377a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:56.650261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.647477 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vxh\" (UniqueName: \"kubernetes.io/projected/078760c3-3213-4e7d-9a60-935413e9377a-kube-api-access-w4vxh\") pod \"payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns\" (UID: \"078760c3-3213-4e7d-9a60-935413e9377a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:56.754693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.754581 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vxh\" (UniqueName: \"kubernetes.io/projected/078760c3-3213-4e7d-9a60-935413e9377a-kube-api-access-w4vxh\") pod \"payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns\" (UID: \"078760c3-3213-4e7d-9a60-935413e9377a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:56.755356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.755303 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078760c3-3213-4e7d-9a60-935413e9377a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns\" (UID: \"078760c3-3213-4e7d-9a60-935413e9377a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:56.756663 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.756639 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078760c3-3213-4e7d-9a60-935413e9377a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns\" (UID: \"078760c3-3213-4e7d-9a60-935413e9377a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:56.830312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.830210 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vxh\" (UniqueName: \"kubernetes.io/projected/078760c3-3213-4e7d-9a60-935413e9377a-kube-api-access-w4vxh\") pod \"payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns\" (UID: \"078760c3-3213-4e7d-9a60-935413e9377a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:56.966335 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:43:56.966294 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05e1f66_6c14_4e01_b5f0_8e449dbf718a.slice/crio-024a64b5fcc76de2e16beb5b52056b333d3e8a26de07e12646d695e9bc37fa95 WatchSource:0}: Error finding container 024a64b5fcc76de2e16beb5b52056b333d3e8a26de07e12646d695e9bc37fa95: Status 404 returned error can't find the container with id 024a64b5fcc76de2e16beb5b52056b333d3e8a26de07e12646d695e9bc37fa95 Oct 30 21:43:56.967318 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:56.967288 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8"] Oct 30 21:43:57.058072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:57.058052 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:57.199456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:57.199416 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" event={"ID":"d05e1f66-6c14-4e01-b5f0-8e449dbf718a","Type":"ContainerStarted","Data":"47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f"} Oct 30 21:43:57.199652 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:57.199510 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:43:57.199652 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:57.199539 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" event={"ID":"d05e1f66-6c14-4e01-b5f0-8e449dbf718a","Type":"ContainerStarted","Data":"024a64b5fcc76de2e16beb5b52056b333d3e8a26de07e12646d695e9bc37fa95"} Oct 30 21:43:57.320119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:57.320073 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" podStartSLOduration=1.320058683 podStartE2EDuration="1.320058683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:57.315206557 +0000 UTC m=+1627.448579756" watchObservedRunningTime="2026-10-30 21:43:57.320058683 +0000 UTC m=+1627.453431941" Oct 30 21:43:57.441640 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:43:57.441606 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078760c3_3213_4e7d_9a60_935413e9377a.slice/crio-79e316e3e197fd6884a4cf31813f2dedff89227dc2cf3d422379abac59c59487 WatchSource:0}: Error finding container 79e316e3e197fd6884a4cf31813f2dedff89227dc2cf3d422379abac59c59487: Status 404 returned error can't find the container with id 79e316e3e197fd6884a4cf31813f2dedff89227dc2cf3d422379abac59c59487 Oct 30 21:43:57.443309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:57.443278 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns"] Oct 30 21:43:58.050416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:58.050384 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n"] Oct 30 21:43:58.055377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:58.055350 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-336f566e-vb55n"] Oct 30 21:43:58.210133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:58.210041 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" event={"ID":"078760c3-3213-4e7d-9a60-935413e9377a","Type":"ContainerStarted","Data":"3d4995ba75df05341eb1f9033fb4ca6b5045239c28517bb05fcf802a58204ca6"} Oct 30 21:43:58.210746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:58.210705 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:43:58.210746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:58.210742 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" event={"ID":"078760c3-3213-4e7d-9a60-935413e9377a","Type":"ContainerStarted","Data":"79e316e3e197fd6884a4cf31813f2dedff89227dc2cf3d422379abac59c59487"} Oct 30 21:43:58.328664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:58.328598 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" podStartSLOduration=2.328580631 podStartE2EDuration="2.328580631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:58.32686439 +0000 UTC m=+1628.460237582" watchObservedRunningTime="2026-10-30 21:43:58.328580631 +0000 UTC m=+1628.461953832" Oct 30 21:43:59.987906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:43:59.987876 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e31e45-a29d-4571-ba6d-b04fe08c9169" path="/var/lib/kubelet/pods/f9e31e45-a29d-4571-ba6d-b04fe08c9169/volumes" Oct 30 21:44:01.050982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:01.050944 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6"] Oct 30 21:44:01.100766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:01.100728 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq"] Oct 30 21:44:01.105339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:01.105323 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09025997-7zmj6"] Oct 30 21:44:01.109803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:01.109775 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1b355-tlnkq"] Oct 30 21:44:01.987891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:01.987854 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d" path="/var/lib/kubelet/pods/6e9daaca-776d-4d69-9cbf-4d96f1d0dd1d/volumes" Oct 30 21:44:01.988412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:01.988398 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14a218f-25ae-4f71-9800-917eb5c993db" path="/var/lib/kubelet/pods/d14a218f-25ae-4f71-9800-917eb5c993db/volumes" Oct 30 21:44:05.058970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:05.058934 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg"] Oct 30 21:44:05.108372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:05.108337 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec10f526-sdlhg"] Oct 30 21:44:05.988114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:05.988078 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97648b85-6605-4a9a-9be1-d4117b648245" path="/var/lib/kubelet/pods/97648b85-6605-4a9a-9be1-d4117b648245/volumes" Oct 30 21:44:07.380714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.380681 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz"] Oct 30 21:44:07.387022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.387000 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:07.439430 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.438507 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz"] Oct 30 21:44:07.575133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.575103 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-29b7ead1-dockercfg-nfld5\"" Oct 30 21:44:07.575339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.575192 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-29b7ead1\"" Oct 30 21:44:07.579543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.577472 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e1c745-932a-423e-9abe-3b4f5b1d7246-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz\" (UID: \"09e1c745-932a-423e-9abe-3b4f5b1d7246\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:07.579543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.577802 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmq9\" (UniqueName: \"kubernetes.io/projected/09e1c745-932a-423e-9abe-3b4f5b1d7246-kube-api-access-spmq9\") pod \"payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz\" (UID: \"09e1c745-932a-423e-9abe-3b4f5b1d7246\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:07.592218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.592168 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8"] Oct 30 21:44:07.597889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.597861 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:07.598052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.597991 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8"] Oct 30 21:44:07.679749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.679640 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e1c745-932a-423e-9abe-3b4f5b1d7246-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz\" (UID: \"09e1c745-932a-423e-9abe-3b4f5b1d7246\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:07.679999 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.679978 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmq9\" (UniqueName: \"kubernetes.io/projected/09e1c745-932a-423e-9abe-3b4f5b1d7246-kube-api-access-spmq9\") pod \"payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz\" (UID: \"09e1c745-932a-423e-9abe-3b4f5b1d7246\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:07.680755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.680730 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e1c745-932a-423e-9abe-3b4f5b1d7246-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz\" (UID: \"09e1c745-932a-423e-9abe-3b4f5b1d7246\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:07.739823 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.739792 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmq9\" (UniqueName: \"kubernetes.io/projected/09e1c745-932a-423e-9abe-3b4f5b1d7246-kube-api-access-spmq9\") pod \"payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz\" (UID: \"09e1c745-932a-423e-9abe-3b4f5b1d7246\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:07.781707 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.781669 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22761589-b83d-4520-901c-3d8eef2b3421-plugins-config-volume\") pod \"payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8\" (UID: \"22761589-b83d-4520-901c-3d8eef2b3421\") " pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:07.781927 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.781802 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbzhb\" (UniqueName: \"kubernetes.io/projected/22761589-b83d-4520-901c-3d8eef2b3421-kube-api-access-lbzhb\") pod \"payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8\" (UID: \"22761589-b83d-4520-901c-3d8eef2b3421\") " pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:07.883589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.883554 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22761589-b83d-4520-901c-3d8eef2b3421-plugins-config-volume\") pod \"payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8\" (UID: \"22761589-b83d-4520-901c-3d8eef2b3421\") " pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:07.885556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.885528 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22761589-b83d-4520-901c-3d8eef2b3421-plugins-config-volume\") pod \"payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8\" (UID: \"22761589-b83d-4520-901c-3d8eef2b3421\") " pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:07.885810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.885791 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbzhb\" (UniqueName: \"kubernetes.io/projected/22761589-b83d-4520-901c-3d8eef2b3421-kube-api-access-lbzhb\") pod \"payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8\" (UID: \"22761589-b83d-4520-901c-3d8eef2b3421\") " pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:07.947414 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:07.947338 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbzhb\" (UniqueName: \"kubernetes.io/projected/22761589-b83d-4520-901c-3d8eef2b3421-kube-api-access-lbzhb\") pod \"payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8\" (UID: \"22761589-b83d-4520-901c-3d8eef2b3421\") " pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:08.002386 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:08.002352 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:08.060194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:08.060161 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh"] Oct 30 21:44:08.080459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:08.066654 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1bb153b-qr2lh"] Oct 30 21:44:08.214042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:08.213950 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:44:08.220277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:08.220253 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:08.429929 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:44:08.429897 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e1c745_932a_423e_9abe_3b4f5b1d7246.slice/crio-f5c44b7d4438c033d108725c537d917fbc0299d94f4645e2ad7becc95739ab9b WatchSource:0}: Error finding container f5c44b7d4438c033d108725c537d917fbc0299d94f4645e2ad7becc95739ab9b: Status 404 returned error can't find the container with id f5c44b7d4438c033d108725c537d917fbc0299d94f4645e2ad7becc95739ab9b Oct 30 21:44:08.431851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:08.431688 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz"] Oct 30 21:44:08.616527 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:44:08.616464 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22761589_b83d_4520_901c_3d8eef2b3421.slice/crio-3c01ab3d70c73d22590291b3f01460fc29d254163d742aef5673525cff14e05c WatchSource:0}: Error finding container 3c01ab3d70c73d22590291b3f01460fc29d254163d742aef5673525cff14e05c: Status 404 returned error can't find the container with id 3c01ab3d70c73d22590291b3f01460fc29d254163d742aef5673525cff14e05c Oct 30 21:44:08.617064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:08.617013 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8"] Oct 30 21:44:09.223194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.223159 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:44:09.353590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.353524 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" event={"ID":"22761589-b83d-4520-901c-3d8eef2b3421","Type":"ContainerStarted","Data":"c6cddec16bfb3b142d36eb9574664e8fc3081c2fd05d726ebb2068088ec2740f"} Oct 30 21:44:09.353760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.353633 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" event={"ID":"22761589-b83d-4520-901c-3d8eef2b3421","Type":"ContainerStarted","Data":"3c01ab3d70c73d22590291b3f01460fc29d254163d742aef5673525cff14e05c"} Oct 30 21:44:09.354067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.354047 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:09.356777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.356745 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" event={"ID":"09e1c745-932a-423e-9abe-3b4f5b1d7246","Type":"ContainerStarted","Data":"cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee"} Oct 30 21:44:09.356939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.356824 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:09.356939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.356849 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" event={"ID":"09e1c745-932a-423e-9abe-3b4f5b1d7246","Type":"ContainerStarted","Data":"f5c44b7d4438c033d108725c537d917fbc0299d94f4645e2ad7becc95739ab9b"} Oct 30 21:44:09.475708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.475597 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" podStartSLOduration=2.475575524 podStartE2EDuration="2.475575524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:09.468762323 +0000 UTC m=+1639.602135523" watchObservedRunningTime="2026-10-30 21:44:09.475575524 +0000 UTC m=+1639.608948734" Oct 30 21:44:09.588527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.588460 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" podStartSLOduration=2.588444906 podStartE2EDuration="2.588444906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:09.583932057 +0000 UTC m=+1639.717305257" watchObservedRunningTime="2026-10-30 21:44:09.588444906 +0000 UTC m=+1639.721818103" Oct 30 21:44:09.988828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:09.988796 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a705f6-736d-42d6-a03c-33000cf92048" path="/var/lib/kubelet/pods/77a705f6-736d-42d6-a03c-33000cf92048/volumes" Oct 30 21:44:12.479384 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:44:12.479350 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:44:20.369475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:20.369444 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:44:20.369925 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:20.369539 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:44:25.050143 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:25.050110 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c"] Oct 30 21:44:25.098029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:25.097999 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428d85a5-fgq2c"] Oct 30 21:44:25.988716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:25.988682 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1" path="/var/lib/kubelet/pods/0f11d78e-9b98-4c72-bdb4-81ec4b31f3b1/volumes" Oct 30 21:44:27.050225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:27.050192 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw"] Oct 30 21:44:27.099090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:27.099058 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3a3ff6-dbqrw"] Oct 30 21:44:27.987317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:27.987281 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75fb46d5-9526-4d36-a533-7974823eae7a" path="/var/lib/kubelet/pods/75fb46d5-9526-4d36-a533-7974823eae7a/volumes" Oct 30 21:44:32.047293 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.047216 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849"] Oct 30 21:44:32.096718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.096685 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-348cabb3-b2849"] Oct 30 21:44:32.367561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.367461 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk"] Oct 30 21:44:32.373419 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.373402 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.431852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.431826 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk"] Oct 30 21:44:32.474404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.474374 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:32.474629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.474447 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:32.474629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.474474 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:32.476192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.476173 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:44:32.533083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.533052 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.533237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.533169 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nrx\" (UniqueName: \"kubernetes.io/projected/591668dd-6f81-40cc-b1fd-13017a67bd72-kube-api-access-l9nrx\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.533289 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.533265 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.634904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.634824 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nrx\" (UniqueName: \"kubernetes.io/projected/591668dd-6f81-40cc-b1fd-13017a67bd72-kube-api-access-l9nrx\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.635076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.634920 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.635146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.635084 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.635789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.635718 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.637522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.637475 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.691296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.691270 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nrx\" (UniqueName: \"kubernetes.io/projected/591668dd-6f81-40cc-b1fd-13017a67bd72-kube-api-access-l9nrx\") pod \"maas-api-revoke-sub-5fb1ff3d-ptwnk\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:32.989119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:32.989047 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:33.376306 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:44:33.376276 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591668dd_6f81_40cc_b1fd_13017a67bd72.slice/crio-a012ff18813b65641cff69e3e21bcc609a5c8556574e4bc57d6683c32d750a7c WatchSource:0}: Error finding container a012ff18813b65641cff69e3e21bcc609a5c8556574e4bc57d6683c32d750a7c: Status 404 returned error can't find the container with id a012ff18813b65641cff69e3e21bcc609a5c8556574e4bc57d6683c32d750a7c Oct 30 21:44:33.376904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:33.376880 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk"] Oct 30 21:44:33.583811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:33.583775 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" event={"ID":"591668dd-6f81-40cc-b1fd-13017a67bd72","Type":"ContainerStarted","Data":"a012ff18813b65641cff69e3e21bcc609a5c8556574e4bc57d6683c32d750a7c"} Oct 30 21:44:33.993719 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:33.993681 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaebcde1-c1ae-4e37-a475-2a48cd0cebdb" path="/var/lib/kubelet/pods/eaebcde1-c1ae-4e37-a475-2a48cd0cebdb/volumes" Oct 30 21:44:34.594689 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:34.594651 4526 generic.go:356] "Generic (PLEG): container finished" podID="591668dd-6f81-40cc-b1fd-13017a67bd72" containerID="9ead3085b9907ccff80995ec3e8cf3c0512d2cae7ec7555bb8a2ee9f0845b4d9" exitCode=0 Oct 30 21:44:34.595123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:34.594799 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" event={"ID":"591668dd-6f81-40cc-b1fd-13017a67bd72","Type":"ContainerDied","Data":"9ead3085b9907ccff80995ec3e8cf3c0512d2cae7ec7555bb8a2ee9f0845b4d9"} Oct 30 21:44:35.782220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.782186 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:35.801829 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.801800 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-openshift-ca\") pod \"591668dd-6f81-40cc-b1fd-13017a67bd72\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " Oct 30 21:44:35.802149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.802127 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-openshift-ca" pod "591668dd-6f81-40cc-b1fd-13017a67bd72" (UID: "591668dd-6f81-40cc-b1fd-13017a67bd72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:35.802149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.802142 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591668dd-6f81-40cc-b1fd-13017a67bd72-kube-api-access-l9nrx\" (UniqueName: \"kubernetes.io/projected/591668dd-6f81-40cc-b1fd-13017a67bd72-kube-api-access-l9nrx\") pod \"591668dd-6f81-40cc-b1fd-13017a67bd72\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " Oct 30 21:44:35.802286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.802185 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-xks-ca\") pod \"591668dd-6f81-40cc-b1fd-13017a67bd72\" (UID: \"591668dd-6f81-40cc-b1fd-13017a67bd72\") " Oct 30 21:44:35.802885 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.802785 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:35.804441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.804410 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-xks-ca" pod "591668dd-6f81-40cc-b1fd-13017a67bd72" (UID: "591668dd-6f81-40cc-b1fd-13017a67bd72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:35.804567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.804450 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591668dd-6f81-40cc-b1fd-13017a67bd72-kube-api-access-l9nrx" pod "591668dd-6f81-40cc-b1fd-13017a67bd72" (UID: "591668dd-6f81-40cc-b1fd-13017a67bd72"). InnerVolumeSpecName "kube-api-access-l9nrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:35.904360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.904272 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9nrx\" (UniqueName: \"kubernetes.io/projected/591668dd-6f81-40cc-b1fd-13017a67bd72-kube-api-access-l9nrx\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:35.904360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:35.904302 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591668dd-6f81-40cc-b1fd-13017a67bd72-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:36.052937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:36.052897 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf"] Oct 30 21:44:36.057631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:36.057607 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793273d-gtxxf"] Oct 30 21:44:36.664948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:36.664916 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" event={"ID":"591668dd-6f81-40cc-b1fd-13017a67bd72","Type":"ContainerDied","Data":"a012ff18813b65641cff69e3e21bcc609a5c8556574e4bc57d6683c32d750a7c"} Oct 30 21:44:36.665158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:36.664973 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a012ff18813b65641cff69e3e21bcc609a5c8556574e4bc57d6683c32d750a7c" Oct 30 21:44:36.665158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:36.665048 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk" Oct 30 21:44:37.050124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:37.050087 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg"] Oct 30 21:44:37.055052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:37.055021 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-591b8451-w9lwg"] Oct 30 21:44:37.996704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:37.996670 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f351c93-77a7-4cbb-8dc5-d27470e54256" path="/var/lib/kubelet/pods/1f351c93-77a7-4cbb-8dc5-d27470e54256/volumes" Oct 30 21:44:37.997141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:37.997128 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b96cf3f-18c8-45f5-a381-ab923fcd5f7b" path="/var/lib/kubelet/pods/5b96cf3f-18c8-45f5-a381-ab923fcd5f7b/volumes" Oct 30 21:44:42.480605 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:44:42.480570 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:44:44.879672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.879639 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr"] Oct 30 21:44:44.880866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.880840 4526 cpu_manager.go:414] "containerMap: removing container" podUID="591668dd-6f81-40cc-b1fd-13017a67bd72" containerName="revoke-keys" Oct 30 21:44:44.881055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.881040 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591668dd-6f81-40cc-b1fd-13017a67bd72" containerName="revoke-keys" Oct 30 21:44:44.881451 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.881419 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="591668dd-6f81-40cc-b1fd-13017a67bd72" containerName="revoke-keys" Oct 30 21:44:44.886071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.886051 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:44.896952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.896880 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5fs\" (UniqueName: \"kubernetes.io/projected/44a5707e-b59f-44f5-b934-03be6da8edea-kube-api-access-bh5fs\") pod \"payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr\" (UID: \"44a5707e-b59f-44f5-b934-03be6da8edea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:44.897048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.897029 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a5707e-b59f-44f5-b934-03be6da8edea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr\" (UID: \"44a5707e-b59f-44f5-b934-03be6da8edea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:44.921363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.919031 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr"] Oct 30 21:44:44.961954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.961922 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-d6a241\"" Oct 30 21:44:44.962517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.962477 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-d6a241-dockercfg-s6dtr\"" Oct 30 21:44:44.998639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.998587 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a5707e-b59f-44f5-b934-03be6da8edea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr\" (UID: \"44a5707e-b59f-44f5-b934-03be6da8edea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:44.999809 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.999462 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5fs\" (UniqueName: \"kubernetes.io/projected/44a5707e-b59f-44f5-b934-03be6da8edea-kube-api-access-bh5fs\") pod \"payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr\" (UID: \"44a5707e-b59f-44f5-b934-03be6da8edea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:44.999809 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:44.999593 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a5707e-b59f-44f5-b934-03be6da8edea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr\" (UID: \"44a5707e-b59f-44f5-b934-03be6da8edea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:45.022258 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.022219 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj"] Oct 30 21:44:45.028400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.028376 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:45.057778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.057743 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5fs\" (UniqueName: \"kubernetes.io/projected/44a5707e-b59f-44f5-b934-03be6da8edea-kube-api-access-bh5fs\") pod \"payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr\" (UID: \"44a5707e-b59f-44f5-b934-03be6da8edea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:45.105034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.104997 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47e4ad1-21dc-448c-b169-6230907371e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj\" (UID: \"b47e4ad1-21dc-448c-b169-6230907371e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:45.105246 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.105072 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lnpv\" (UniqueName: \"kubernetes.io/projected/b47e4ad1-21dc-448c-b169-6230907371e9-kube-api-access-8lnpv\") pod \"payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj\" (UID: \"b47e4ad1-21dc-448c-b169-6230907371e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:45.118111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.118064 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj"] Oct 30 21:44:45.220595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.220445 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:45.232552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.222025 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47e4ad1-21dc-448c-b169-6230907371e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj\" (UID: \"b47e4ad1-21dc-448c-b169-6230907371e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:45.232552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.222834 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47e4ad1-21dc-448c-b169-6230907371e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj\" (UID: \"b47e4ad1-21dc-448c-b169-6230907371e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:45.232552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.222981 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lnpv\" (UniqueName: \"kubernetes.io/projected/b47e4ad1-21dc-448c-b169-6230907371e9-kube-api-access-8lnpv\") pod \"payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj\" (UID: \"b47e4ad1-21dc-448c-b169-6230907371e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:45.342513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.334912 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lnpv\" (UniqueName: \"kubernetes.io/projected/b47e4ad1-21dc-448c-b169-6230907371e9-kube-api-access-8lnpv\") pod \"payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj\" (UID: \"b47e4ad1-21dc-448c-b169-6230907371e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:45.344896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.344865 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:45.729079 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:44:45.729049 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a5707e_b59f_44f5_b934_03be6da8edea.slice/crio-17d23ce425335a57b6122acfae3f29b76ffcac3af4223d83a2cf181e38fb9ecd WatchSource:0}: Error finding container 17d23ce425335a57b6122acfae3f29b76ffcac3af4223d83a2cf181e38fb9ecd: Status 404 returned error can't find the container with id 17d23ce425335a57b6122acfae3f29b76ffcac3af4223d83a2cf181e38fb9ecd Oct 30 21:44:45.729219 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.729121 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr"] Oct 30 21:44:45.748658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.748606 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" event={"ID":"44a5707e-b59f-44f5-b934-03be6da8edea","Type":"ContainerStarted","Data":"17d23ce425335a57b6122acfae3f29b76ffcac3af4223d83a2cf181e38fb9ecd"} Oct 30 21:44:45.788463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:45.788432 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj"] Oct 30 21:44:45.794886 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:44:45.794861 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47e4ad1_21dc_448c_b169_6230907371e9.slice/crio-7b7e68428b07c0ed20f83bd7183fe92fdb9342892871ff81a37f826c0368ab0e WatchSource:0}: Error finding container 7b7e68428b07c0ed20f83bd7183fe92fdb9342892871ff81a37f826c0368ab0e: Status 404 returned error can't find the container with id 7b7e68428b07c0ed20f83bd7183fe92fdb9342892871ff81a37f826c0368ab0e Oct 30 21:44:46.048645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:46.048602 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw"] Oct 30 21:44:46.053063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:46.053042 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9597406-9drzw"] Oct 30 21:44:46.758688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:46.758644 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" event={"ID":"44a5707e-b59f-44f5-b934-03be6da8edea","Type":"ContainerStarted","Data":"da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70"} Oct 30 21:44:46.758898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:46.758713 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:46.760462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:46.760441 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" event={"ID":"b47e4ad1-21dc-448c-b169-6230907371e9","Type":"ContainerStarted","Data":"1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113"} Oct 30 21:44:46.760625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:46.760479 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" event={"ID":"b47e4ad1-21dc-448c-b169-6230907371e9","Type":"ContainerStarted","Data":"7b7e68428b07c0ed20f83bd7183fe92fdb9342892871ff81a37f826c0368ab0e"} Oct 30 21:44:46.760625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:46.760551 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:44:46.891578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:46.891529 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" podStartSLOduration=2.891514368 podStartE2EDuration="2.891514368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:46.888569305 +0000 UTC m=+1677.021942505" watchObservedRunningTime="2026-10-30 21:44:46.891514368 +0000 UTC m=+1677.024887565" Oct 30 21:44:47.009563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:47.009396 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" podStartSLOduration=3.009371687 podStartE2EDuration="3.009371687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:47.003110117 +0000 UTC m=+1677.136483320" watchObservedRunningTime="2026-10-30 21:44:47.009371687 +0000 UTC m=+1677.142744886" Oct 30 21:44:47.049978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:47.049935 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt"] Oct 30 21:44:47.055796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:47.055762 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9f3038e-rk8bt"] Oct 30 21:44:47.987355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:47.987321 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4215fa7d-c8cc-47f3-a8f8-b7d80b64418d" path="/var/lib/kubelet/pods/4215fa7d-c8cc-47f3-a8f8-b7d80b64418d/volumes" Oct 30 21:44:47.987932 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:47.987916 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674fa73b-55ac-4158-a819-83483a19fb32" path="/var/lib/kubelet/pods/674fa73b-55ac-4158-a819-83483a19fb32/volumes" Oct 30 21:44:48.056656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:48.056614 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh"] Oct 30 21:44:48.106503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:48.106457 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw"] Oct 30 21:44:48.111053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:48.111030 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f93ecb-v5rlh"] Oct 30 21:44:48.115324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:48.115303 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85dc19fa-56zzw"] Oct 30 21:44:49.098765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.098731 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf"] Oct 30 21:44:49.104346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.104327 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.161261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.161231 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf"] Oct 30 21:44:49.204727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.204689 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:49.204727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.204703 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:44:49.204948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.204747 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:44:49.206625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.206603 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:44:49.212596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.212568 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz6lw\" (UniqueName: \"kubernetes.io/projected/e0dc8bad-3f60-4fbe-bac4-006595417e45-kube-api-access-rz6lw\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.212733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.212625 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.212821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.212798 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.314801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.314766 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz6lw\" (UniqueName: \"kubernetes.io/projected/e0dc8bad-3f60-4fbe-bac4-006595417e45-kube-api-access-rz6lw\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.314989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.314826 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.314989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.314893 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.315629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.315560 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.317377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.317357 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.373632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.373562 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz6lw\" (UniqueName: \"kubernetes.io/projected/e0dc8bad-3f60-4fbe-bac4-006595417e45-kube-api-access-rz6lw\") pod \"maas-api-revoke-keys-e2e-ait-9173600a-s88zf\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.426312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.426280 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:49.816299 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:44:49.816266 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dc8bad_3f60_4fbe_bac4_006595417e45.slice/crio-0fc8f9e867dd5e3f18a521b929dad4865cbddb6b1b27793b9636deaa0cc6e589 WatchSource:0}: Error finding container 0fc8f9e867dd5e3f18a521b929dad4865cbddb6b1b27793b9636deaa0cc6e589: Status 404 returned error can't find the container with id 0fc8f9e867dd5e3f18a521b929dad4865cbddb6b1b27793b9636deaa0cc6e589 Oct 30 21:44:49.818200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.818160 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf"] Oct 30 21:44:49.988659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.988615 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3ee4a0-d451-4b8b-8586-da08c01d8321" path="/var/lib/kubelet/pods/0a3ee4a0-d451-4b8b-8586-da08c01d8321/volumes" Oct 30 21:44:49.989201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:49.989176 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bb8bee-a24f-479b-8331-bca0f146a40d" path="/var/lib/kubelet/pods/31bb8bee-a24f-479b-8331-bca0f146a40d/volumes" Oct 30 21:44:50.048402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.048368 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz"] Oct 30 21:44:50.052689 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.052661 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41819b1d-dnvjz"] Oct 30 21:44:50.807387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.806854 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" event={"ID":"e0dc8bad-3f60-4fbe-bac4-006595417e45","Type":"ContainerStarted","Data":"8b43f1ccc8c3f5fbd6e07ad01a658534a62a8955e75e3e96736691a9c8754c36"} Oct 30 21:44:50.807387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.806929 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" event={"ID":"e0dc8bad-3f60-4fbe-bac4-006595417e45","Type":"ContainerStarted","Data":"0fc8f9e867dd5e3f18a521b929dad4865cbddb6b1b27793b9636deaa0cc6e589"} Oct 30 21:44:50.871831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.871802 4526 scope.go:122] "RemoveContainer" containerID="b9499d9ebe29eaba174d6a1ecfdd0064b9e8bd0998db3c4b4777ec6778063c44" Oct 30 21:44:50.883405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.883377 4526 scope.go:122] "RemoveContainer" containerID="f1ff63d614f341fc1d08927a86bacbb1fa39b38ceb0e4cef88ee0a519f301349" Oct 30 21:44:50.893890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.893874 4526 scope.go:122] "RemoveContainer" containerID="2ff20305195b717a32b7d45798fab29c518802e41304d1607bbf5c465c63fe50" Oct 30 21:44:50.904294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.904274 4526 scope.go:122] "RemoveContainer" containerID="eb9483c2c12ab604522ab27b34b2e24dd22222d1a3cf6f5fca2d642f77c9bdd4" Oct 30 21:44:50.914692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.914672 4526 scope.go:122] "RemoveContainer" containerID="0d677012f1f8aa6c5247df8f0523a552f2e942eef1a8b43735f18e63fbaeeb95" Oct 30 21:44:50.923000 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.922947 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" podStartSLOduration=1.92293345 podStartE2EDuration="1.92293345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:50.919774857 +0000 UTC m=+1681.053148061" watchObservedRunningTime="2026-10-30 21:44:50.92293345 +0000 UTC m=+1681.056306648" Oct 30 21:44:50.925452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.925329 4526 scope.go:122] "RemoveContainer" containerID="f2a05687bc39bd25da7e90708106d687aac13c36a3eb2fb5ff96e98adf29bc5d" Oct 30 21:44:50.935226 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.935205 4526 scope.go:122] "RemoveContainer" containerID="fbf865f476ecb037cfe7aa873ec36362f4ca4ed1526755718cfc57bda37522f3" Oct 30 21:44:50.945662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:50.945645 4526 scope.go:122] "RemoveContainer" containerID="ec585ff982b8e41c634cbfc122baca3536ff3e1f4eafdae1378d614df43fe77c" Oct 30 21:44:51.817088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:51.817052 4526 generic.go:356] "Generic (PLEG): container finished" podID="e0dc8bad-3f60-4fbe-bac4-006595417e45" containerID="8b43f1ccc8c3f5fbd6e07ad01a658534a62a8955e75e3e96736691a9c8754c36" exitCode=0 Oct 30 21:44:51.817580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:51.817134 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" event={"ID":"e0dc8bad-3f60-4fbe-bac4-006595417e45","Type":"ContainerDied","Data":"8b43f1ccc8c3f5fbd6e07ad01a658534a62a8955e75e3e96736691a9c8754c36"} Oct 30 21:44:51.988093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:51.988063 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2b5c52-ab4a-41cd-a050-320135ef3271" path="/var/lib/kubelet/pods/7d2b5c52-ab4a-41cd-a050-320135ef3271/volumes" Oct 30 21:44:52.996903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:52.996874 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:53.081601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.081510 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-xks-ca\") pod \"e0dc8bad-3f60-4fbe-bac4-006595417e45\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " Oct 30 21:44:53.081760 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.081630 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0dc8bad-3f60-4fbe-bac4-006595417e45-kube-api-access-rz6lw\" (UniqueName: \"kubernetes.io/projected/e0dc8bad-3f60-4fbe-bac4-006595417e45-kube-api-access-rz6lw\") pod \"e0dc8bad-3f60-4fbe-bac4-006595417e45\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " Oct 30 21:44:53.081873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.081855 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-openshift-ca\") pod \"e0dc8bad-3f60-4fbe-bac4-006595417e45\" (UID: \"e0dc8bad-3f60-4fbe-bac4-006595417e45\") " Oct 30 21:44:53.082190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.082159 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-openshift-ca" pod "e0dc8bad-3f60-4fbe-bac4-006595417e45" (UID: "e0dc8bad-3f60-4fbe-bac4-006595417e45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:53.082584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.082564 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.084128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.084100 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dc8bad-3f60-4fbe-bac4-006595417e45-kube-api-access-rz6lw" pod "e0dc8bad-3f60-4fbe-bac4-006595417e45" (UID: "e0dc8bad-3f60-4fbe-bac4-006595417e45"). InnerVolumeSpecName "kube-api-access-rz6lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:53.084244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.084132 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-xks-ca" pod "e0dc8bad-3f60-4fbe-bac4-006595417e45" (UID: "e0dc8bad-3f60-4fbe-bac4-006595417e45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:53.184384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.184349 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dc8bad-3f60-4fbe-bac4-006595417e45-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.184384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.184381 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz6lw\" (UniqueName: \"kubernetes.io/projected/e0dc8bad-3f60-4fbe-bac4-006595417e45-kube-api-access-rz6lw\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:53.886479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.886373 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" event={"ID":"e0dc8bad-3f60-4fbe-bac4-006595417e45","Type":"ContainerDied","Data":"0fc8f9e867dd5e3f18a521b929dad4865cbddb6b1b27793b9636deaa0cc6e589"} Oct 30 21:44:53.886479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.886440 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc8f9e867dd5e3f18a521b929dad4865cbddb6b1b27793b9636deaa0cc6e589" Oct 30 21:44:53.887560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:53.886860 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf" Oct 30 21:44:57.773839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:57.773803 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:44:57.774348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:44:57.773953 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:45:00.050983 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.050945 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn"] Oct 30 21:45:00.099640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.099601 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663fb99c-57dsn"] Oct 30 21:45:00.225041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.225010 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr"] Oct 30 21:45:00.225840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.225820 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e0dc8bad-3f60-4fbe-bac4-006595417e45" containerName="revoke-keys" Oct 30 21:45:00.225840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.225838 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0dc8bad-3f60-4fbe-bac4-006595417e45" containerName="revoke-keys" Oct 30 21:45:00.225981 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.225966 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0dc8bad-3f60-4fbe-bac4-006595417e45" containerName="revoke-keys" Oct 30 21:45:00.230139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.230117 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:00.250361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.250324 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd16d13-ff85-4eb2-a567-2800b627aa54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr\" (UID: \"0cd16d13-ff85-4eb2-a567-2800b627aa54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:00.250686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.250652 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjrt2\" (UniqueName: \"kubernetes.io/projected/0cd16d13-ff85-4eb2-a567-2800b627aa54-kube-api-access-kjrt2\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr\" (UID: \"0cd16d13-ff85-4eb2-a567-2800b627aa54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:00.257111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.257074 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs"] Oct 30 21:45:00.262046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.262018 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:00.267764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.267738 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv"] Oct 30 21:45:00.272844 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.272818 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:00.289612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.289581 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq"] Oct 30 21:45:00.295168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.295143 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:00.295613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.295590 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr"] Oct 30 21:45:00.302272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.302205 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:00.311935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.311911 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:45:00.314255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.312699 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:00.319268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.319234 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v"] Oct 30 21:45:00.325049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.325026 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:00.348325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.348092 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h"] Oct 30 21:45:00.353099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.352464 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd16d13-ff85-4eb2-a567-2800b627aa54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr\" (UID: \"0cd16d13-ff85-4eb2-a567-2800b627aa54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:00.353099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.352566 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6vk\" (UniqueName: \"kubernetes.io/projected/acdef6a3-2a30-4bbe-ab10-64abc1239a66-kube-api-access-dd6vk\") pod \"maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv\" (UID: \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:00.353099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.352647 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqkj\" (UniqueName: \"kubernetes.io/projected/1f63cc29-b531-4c4f-af3e-81a3704576ff-kube-api-access-rlqkj\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v\" (UID: \"1f63cc29-b531-4c4f-af3e-81a3704576ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:00.353099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.352895 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f63cc29-b531-4c4f-af3e-81a3704576ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v\" (UID: \"1f63cc29-b531-4c4f-af3e-81a3704576ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:00.353099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.352943 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdef6a3-2a30-4bbe-ab10-64abc1239a66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv\" (UID: \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:00.353099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.352994 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d44537-c273-47c3-b64e-b3225d200495-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq\" (UID: \"a7d44537-c273-47c3-b64e-b3225d200495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:00.354454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.353614 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrt2\" (UniqueName: \"kubernetes.io/projected/0cd16d13-ff85-4eb2-a567-2800b627aa54-kube-api-access-kjrt2\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr\" (UID: \"0cd16d13-ff85-4eb2-a567-2800b627aa54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:00.354454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.353693 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7sc4\" (UniqueName: \"kubernetes.io/projected/a8d299b1-ab44-421a-b922-bacef89a9084-kube-api-access-z7sc4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs\" (UID: \"a8d299b1-ab44-421a-b922-bacef89a9084\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:00.354454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.353803 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d299b1-ab44-421a-b922-bacef89a9084-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs\" (UID: \"a8d299b1-ab44-421a-b922-bacef89a9084\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:00.354454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.353834 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhkvq\" (UniqueName: \"kubernetes.io/projected/a7d44537-c273-47c3-b64e-b3225d200495-kube-api-access-jhkvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq\" (UID: \"a7d44537-c273-47c3-b64e-b3225d200495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:00.355216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.355107 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd16d13-ff85-4eb2-a567-2800b627aa54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr\" (UID: \"0cd16d13-ff85-4eb2-a567-2800b627aa54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:00.367043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.359766 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:00.367043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.360667 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs"] Oct 30 21:45:00.369235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.369203 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v"] Oct 30 21:45:00.369906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.369882 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:00.383366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.379378 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs"] Oct 30 21:45:00.383366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.379613 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:00.390129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.390099 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv"] Oct 30 21:45:00.395886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.395856 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq"] Oct 30 21:45:00.409569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.409538 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v"] Oct 30 21:45:00.416559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.416530 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h"] Oct 30 21:45:00.420646 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.420611 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrt2\" (UniqueName: \"kubernetes.io/projected/0cd16d13-ff85-4eb2-a567-2800b627aa54-kube-api-access-kjrt2\") pod \"maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr\" (UID: \"0cd16d13-ff85-4eb2-a567-2800b627aa54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:00.421818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.421794 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs"] Oct 30 21:45:00.453567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.453535 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v"] Oct 30 21:45:00.455512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.455464 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f63cc29-b531-4c4f-af3e-81a3704576ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v\" (UID: \"1f63cc29-b531-4c4f-af3e-81a3704576ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:00.455747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.455723 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwfh\" (UniqueName: \"kubernetes.io/projected/17547078-dc7b-473d-ac5e-1cd327ffac61-kube-api-access-7zwfh\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h\" (UID: \"17547078-dc7b-473d-ac5e-1cd327ffac61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:00.455911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.455890 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdef6a3-2a30-4bbe-ab10-64abc1239a66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv\" (UID: \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:00.456052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456036 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d44537-c273-47c3-b64e-b3225d200495-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq\" (UID: \"a7d44537-c273-47c3-b64e-b3225d200495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:00.456187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456170 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7sc4\" (UniqueName: \"kubernetes.io/projected/a8d299b1-ab44-421a-b922-bacef89a9084-kube-api-access-z7sc4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs\" (UID: \"a8d299b1-ab44-421a-b922-bacef89a9084\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:00.456328 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456313 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmztx\" (UniqueName: \"kubernetes.io/projected/35eb2ffd-c29e-494a-8aa9-db939e290075-kube-api-access-mmztx\") pod \"maas-api-key-cleanup-29889945-22qvs\" (UID: \"35eb2ffd-c29e-494a-8aa9-db939e290075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:00.456503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456470 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d299b1-ab44-421a-b922-bacef89a9084-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs\" (UID: \"a8d299b1-ab44-421a-b922-bacef89a9084\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:00.456655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456630 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkvq\" (UniqueName: \"kubernetes.io/projected/a7d44537-c273-47c3-b64e-b3225d200495-kube-api-access-jhkvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq\" (UID: \"a7d44537-c273-47c3-b64e-b3225d200495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:00.456846 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456829 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6vk\" (UniqueName: \"kubernetes.io/projected/acdef6a3-2a30-4bbe-ab10-64abc1239a66-kube-api-access-dd6vk\") pod \"maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv\" (UID: \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:00.456955 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456228 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f63cc29-b531-4c4f-af3e-81a3704576ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v\" (UID: \"1f63cc29-b531-4c4f-af3e-81a3704576ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:00.457088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.457061 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17547078-dc7b-473d-ac5e-1cd327ffac61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h\" (UID: \"17547078-dc7b-473d-ac5e-1cd327ffac61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:00.457172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.457117 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqkj\" (UniqueName: \"kubernetes.io/projected/1f63cc29-b531-4c4f-af3e-81a3704576ff-kube-api-access-rlqkj\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v\" (UID: \"1f63cc29-b531-4c4f-af3e-81a3704576ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:00.457172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.457149 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35eb2ffd-c29e-494a-8aa9-db939e290075-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-22qvs\" (UID: \"35eb2ffd-c29e-494a-8aa9-db939e290075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:00.457380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456876 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d44537-c273-47c3-b64e-b3225d200495-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq\" (UID: \"a7d44537-c273-47c3-b64e-b3225d200495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:00.457480 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.457373 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafeeafb-8b05-4378-bb7f-102784b2d91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v\" (UID: \"cafeeafb-8b05-4378-bb7f-102784b2d91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:00.457682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.457653 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72h2q\" (UniqueName: \"kubernetes.io/projected/cafeeafb-8b05-4378-bb7f-102784b2d91b-kube-api-access-72h2q\") pod \"maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v\" (UID: \"cafeeafb-8b05-4378-bb7f-102784b2d91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:00.457882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.456671 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdef6a3-2a30-4bbe-ab10-64abc1239a66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv\" (UID: \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:00.457995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.457535 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d299b1-ab44-421a-b922-bacef89a9084-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs\" (UID: \"a8d299b1-ab44-421a-b922-bacef89a9084\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:00.514162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.514128 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7sc4\" (UniqueName: \"kubernetes.io/projected/a8d299b1-ab44-421a-b922-bacef89a9084-kube-api-access-z7sc4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs\" (UID: \"a8d299b1-ab44-421a-b922-bacef89a9084\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:00.514475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.514457 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkvq\" (UniqueName: \"kubernetes.io/projected/a7d44537-c273-47c3-b64e-b3225d200495-kube-api-access-jhkvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq\" (UID: \"a7d44537-c273-47c3-b64e-b3225d200495\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:00.514578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.514473 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6vk\" (UniqueName: \"kubernetes.io/projected/acdef6a3-2a30-4bbe-ab10-64abc1239a66-kube-api-access-dd6vk\") pod \"maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv\" (UID: \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:00.514643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.514622 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqkj\" (UniqueName: \"kubernetes.io/projected/1f63cc29-b531-4c4f-af3e-81a3704576ff-kube-api-access-rlqkj\") pod \"maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v\" (UID: \"1f63cc29-b531-4c4f-af3e-81a3704576ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:00.547132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.547100 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:00.559465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.559396 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17547078-dc7b-473d-ac5e-1cd327ffac61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h\" (UID: \"17547078-dc7b-473d-ac5e-1cd327ffac61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:00.559465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.559440 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35eb2ffd-c29e-494a-8aa9-db939e290075-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-22qvs\" (UID: \"35eb2ffd-c29e-494a-8aa9-db939e290075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:00.559682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.559581 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafeeafb-8b05-4378-bb7f-102784b2d91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v\" (UID: \"cafeeafb-8b05-4378-bb7f-102784b2d91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:00.559682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.559612 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72h2q\" (UniqueName: \"kubernetes.io/projected/cafeeafb-8b05-4378-bb7f-102784b2d91b-kube-api-access-72h2q\") pod \"maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v\" (UID: \"cafeeafb-8b05-4378-bb7f-102784b2d91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:00.559824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.559685 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwfh\" (UniqueName: \"kubernetes.io/projected/17547078-dc7b-473d-ac5e-1cd327ffac61-kube-api-access-7zwfh\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h\" (UID: \"17547078-dc7b-473d-ac5e-1cd327ffac61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:00.559824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.559785 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmztx\" (UniqueName: \"kubernetes.io/projected/35eb2ffd-c29e-494a-8aa9-db939e290075-kube-api-access-mmztx\") pod \"maas-api-key-cleanup-29889945-22qvs\" (UID: \"35eb2ffd-c29e-494a-8aa9-db939e290075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:00.560277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.560254 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35eb2ffd-c29e-494a-8aa9-db939e290075-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-22qvs\" (UID: \"35eb2ffd-c29e-494a-8aa9-db939e290075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:00.560370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.560272 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17547078-dc7b-473d-ac5e-1cd327ffac61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h\" (UID: \"17547078-dc7b-473d-ac5e-1cd327ffac61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:00.560370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.560338 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafeeafb-8b05-4378-bb7f-102784b2d91b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v\" (UID: \"cafeeafb-8b05-4378-bb7f-102784b2d91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:00.578542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.578511 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:00.588358 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.588330 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:00.618279 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.618201 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwfh\" (UniqueName: \"kubernetes.io/projected/17547078-dc7b-473d-ac5e-1cd327ffac61-kube-api-access-7zwfh\") pod \"maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h\" (UID: \"17547078-dc7b-473d-ac5e-1cd327ffac61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:00.619720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.619586 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmztx\" (UniqueName: \"kubernetes.io/projected/35eb2ffd-c29e-494a-8aa9-db939e290075-kube-api-access-mmztx\") pod \"maas-api-key-cleanup-29889945-22qvs\" (UID: \"35eb2ffd-c29e-494a-8aa9-db939e290075\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:00.621385 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.621360 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72h2q\" (UniqueName: \"kubernetes.io/projected/cafeeafb-8b05-4378-bb7f-102784b2d91b-kube-api-access-72h2q\") pod \"maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v\" (UID: \"cafeeafb-8b05-4378-bb7f-102784b2d91b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:00.631861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.631823 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:00.665632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.663884 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:00.718178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.718109 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:00.721206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.721136 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:00.776231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.776196 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:00.945260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.945227 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr"] Oct 30 21:45:00.952720 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:45:00.952626 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd16d13_ff85_4eb2_a567_2800b627aa54.slice/crio-e61f1d11cc0e66e99b63c88dde14f3bac6c1215e5771e07b6ec1df90534b547c WatchSource:0}: Error finding container e61f1d11cc0e66e99b63c88dde14f3bac6c1215e5771e07b6ec1df90534b547c: Status 404 returned error can't find the container with id e61f1d11cc0e66e99b63c88dde14f3bac6c1215e5771e07b6ec1df90534b547c Oct 30 21:45:00.976583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:00.976540 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" event={"ID":"0cd16d13-ff85-4eb2-a567-2800b627aa54","Type":"ContainerStarted","Data":"e61f1d11cc0e66e99b63c88dde14f3bac6c1215e5771e07b6ec1df90534b547c"} Oct 30 21:45:01.096243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.096215 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq"] Oct 30 21:45:01.103379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.103352 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv"] Oct 30 21:45:01.115563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.114760 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v"] Oct 30 21:45:01.140939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.140911 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs"] Oct 30 21:45:01.167176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.167113 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h"] Oct 30 21:45:01.199060 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:45:01.199006 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f63cc29_b531_4c4f_af3e_81a3704576ff.slice/crio-0486dd21e514ee6d452808c08ee40bd25a995363dff33fe4c44ff9a949ae423a WatchSource:0}: Error finding container 0486dd21e514ee6d452808c08ee40bd25a995363dff33fe4c44ff9a949ae423a: Status 404 returned error can't find the container with id 0486dd21e514ee6d452808c08ee40bd25a995363dff33fe4c44ff9a949ae423a Oct 30 21:45:01.206391 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:45:01.206357 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d299b1_ab44_421a_b922_bacef89a9084.slice/crio-d3cfed769a6f1256b6d68ea3a33b46514d0f85a6dc5aab5fec5cc77d81f53797 WatchSource:0}: Error finding container d3cfed769a6f1256b6d68ea3a33b46514d0f85a6dc5aab5fec5cc77d81f53797: Status 404 returned error can't find the container with id d3cfed769a6f1256b6d68ea3a33b46514d0f85a6dc5aab5fec5cc77d81f53797 Oct 30 21:45:01.295782 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.285899 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v"] Oct 30 21:45:01.381553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.381523 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs"] Oct 30 21:45:01.400771 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:45:01.400726 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafeeafb_8b05_4378_bb7f_102784b2d91b.slice/crio-98b1b0764f92ee1bf6361dd9468d5a577f8e8aafca2fb2033676b2fc1fd93d39 WatchSource:0}: Error finding container 98b1b0764f92ee1bf6361dd9468d5a577f8e8aafca2fb2033676b2fc1fd93d39: Status 404 returned error can't find the container with id 98b1b0764f92ee1bf6361dd9468d5a577f8e8aafca2fb2033676b2fc1fd93d39 Oct 30 21:45:01.988748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.988670 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a1e85d-2b35-4222-bcfa-80d27a294234" path="/var/lib/kubelet/pods/21a1e85d-2b35-4222-bcfa-80d27a294234/volumes" Oct 30 21:45:01.989131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.989106 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" event={"ID":"1f63cc29-b531-4c4f-af3e-81a3704576ff","Type":"ContainerStarted","Data":"0486dd21e514ee6d452808c08ee40bd25a995363dff33fe4c44ff9a949ae423a"} Oct 30 21:45:01.990226 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.990198 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" event={"ID":"0cd16d13-ff85-4eb2-a567-2800b627aa54","Type":"ContainerStarted","Data":"6b75721fa3d4a811bffe563ff6d46c276541e83028f3e47bb53e497d3ea6fef2"} Oct 30 21:45:01.991922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.991895 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" event={"ID":"acdef6a3-2a30-4bbe-ab10-64abc1239a66","Type":"ContainerStarted","Data":"9dd18c51d3eb396c31f775ef955d0d1de73f2847892aada36353cbabc76e42e5"} Oct 30 21:45:01.992037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.991943 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" event={"ID":"acdef6a3-2a30-4bbe-ab10-64abc1239a66","Type":"ContainerStarted","Data":"97c81409b9cf92ea214611ab98716b3dd3bc33f6bdf91e1042fc74ddab31d316"} Oct 30 21:45:01.993339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.993308 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" event={"ID":"17547078-dc7b-473d-ac5e-1cd327ffac61","Type":"ContainerStarted","Data":"de7e1980ccf7e33a3d6779e2291646d2776ce5be16336119d1e36c3499f5a044"} Oct 30 21:45:01.994837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.994812 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" event={"ID":"35eb2ffd-c29e-494a-8aa9-db939e290075","Type":"ContainerStarted","Data":"9b185c79251d26faf1b703de219513627ec942827375fda1b993d8c0b2a7c0a3"} Oct 30 21:45:01.999560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.999535 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" event={"ID":"a8d299b1-ab44-421a-b922-bacef89a9084","Type":"ContainerStarted","Data":"2ff7d7e46d59a3a6b86c4493157ace25c9bf517176b888ba2198cd7bfb887d91"} Oct 30 21:45:01.999877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:01.999855 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" event={"ID":"a8d299b1-ab44-421a-b922-bacef89a9084","Type":"ContainerStarted","Data":"d3cfed769a6f1256b6d68ea3a33b46514d0f85a6dc5aab5fec5cc77d81f53797"} Oct 30 21:45:02.001096 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:02.001074 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" event={"ID":"cafeeafb-8b05-4378-bb7f-102784b2d91b","Type":"ContainerStarted","Data":"98b1b0764f92ee1bf6361dd9468d5a577f8e8aafca2fb2033676b2fc1fd93d39"} Oct 30 21:45:02.002709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:02.002689 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" event={"ID":"a7d44537-c273-47c3-b64e-b3225d200495","Type":"ContainerStarted","Data":"6d0319ddaf968187065bd8976693784fe4126dd89ec50658c2d420f03a163c9d"} Oct 30 21:45:02.002786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:02.002723 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" event={"ID":"a7d44537-c273-47c3-b64e-b3225d200495","Type":"ContainerStarted","Data":"1ae265306056a2c1db2723d050531628d1ec5f4c18241146c59e2cc60acee724"} Oct 30 21:45:02.107742 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:02.107641 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" podStartSLOduration=2.107622718 podStartE2EDuration="2.107622718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.103145793 +0000 UTC m=+1692.236518991" watchObservedRunningTime="2026-10-30 21:45:02.107622718 +0000 UTC m=+1692.240995916" Oct 30 21:45:02.216687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:02.216626 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" podStartSLOduration=2.216610629 podStartE2EDuration="2.216610629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.21557225 +0000 UTC m=+1692.348945448" watchObservedRunningTime="2026-10-30 21:45:02.216610629 +0000 UTC m=+1692.349983828" Oct 30 21:45:02.333223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:02.333167 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" podStartSLOduration=2.333151167 podStartE2EDuration="2.333151167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.328598757 +0000 UTC m=+1692.461971953" watchObservedRunningTime="2026-10-30 21:45:02.333151167 +0000 UTC m=+1692.466524418" Oct 30 21:45:02.462168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:02.460714 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" podStartSLOduration=2.460691994 podStartE2EDuration="2.460691994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.445918595 +0000 UTC m=+1692.579291794" watchObservedRunningTime="2026-10-30 21:45:02.460691994 +0000 UTC m=+1692.594065192" Oct 30 21:45:03.014844 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.014756 4526 generic.go:356] "Generic (PLEG): container finished" podID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerID="6b75721fa3d4a811bffe563ff6d46c276541e83028f3e47bb53e497d3ea6fef2" exitCode=60 Oct 30 21:45:03.015019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.014845 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" event={"ID":"0cd16d13-ff85-4eb2-a567-2800b627aa54","Type":"ContainerDied","Data":"6b75721fa3d4a811bffe563ff6d46c276541e83028f3e47bb53e497d3ea6fef2"} Oct 30 21:45:03.015200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.015185 4526 scope.go:122] "RemoveContainer" containerID="6b75721fa3d4a811bffe563ff6d46c276541e83028f3e47bb53e497d3ea6fef2" Oct 30 21:45:03.017259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.017210 4526 generic.go:356] "Generic (PLEG): container finished" podID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerID="9dd18c51d3eb396c31f775ef955d0d1de73f2847892aada36353cbabc76e42e5" exitCode=60 Oct 30 21:45:03.017259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.017245 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" event={"ID":"acdef6a3-2a30-4bbe-ab10-64abc1239a66","Type":"ContainerDied","Data":"9dd18c51d3eb396c31f775ef955d0d1de73f2847892aada36353cbabc76e42e5"} Oct 30 21:45:03.017633 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.017619 4526 scope.go:122] "RemoveContainer" containerID="9dd18c51d3eb396c31f775ef955d0d1de73f2847892aada36353cbabc76e42e5" Oct 30 21:45:03.019341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.019311 4526 generic.go:356] "Generic (PLEG): container finished" podID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerID="30fefa1a603d2e2858b4d8d1740e0e14b2758ae63af6e1357b96b71f0f6ecba5" exitCode=60 Oct 30 21:45:03.019766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.019391 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" event={"ID":"17547078-dc7b-473d-ac5e-1cd327ffac61","Type":"ContainerDied","Data":"30fefa1a603d2e2858b4d8d1740e0e14b2758ae63af6e1357b96b71f0f6ecba5"} Oct 30 21:45:03.021129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.020956 4526 scope.go:122] "RemoveContainer" containerID="30fefa1a603d2e2858b4d8d1740e0e14b2758ae63af6e1357b96b71f0f6ecba5" Oct 30 21:45:03.021510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.021431 4526 generic.go:356] "Generic (PLEG): container finished" podID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerID="7c78f3ecc67501299ed4662ac2ea6997ef61f82160f39299690b3bc668115286" exitCode=60 Oct 30 21:45:03.021567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.021503 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" event={"ID":"35eb2ffd-c29e-494a-8aa9-db939e290075","Type":"ContainerDied","Data":"7c78f3ecc67501299ed4662ac2ea6997ef61f82160f39299690b3bc668115286"} Oct 30 21:45:03.021785 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.021769 4526 scope.go:122] "RemoveContainer" containerID="7c78f3ecc67501299ed4662ac2ea6997ef61f82160f39299690b3bc668115286" Oct 30 21:45:03.023364 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.023345 4526 generic.go:356] "Generic (PLEG): container finished" podID="a8d299b1-ab44-421a-b922-bacef89a9084" containerID="2ff7d7e46d59a3a6b86c4493157ace25c9bf517176b888ba2198cd7bfb887d91" exitCode=60 Oct 30 21:45:03.023444 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.023407 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" event={"ID":"a8d299b1-ab44-421a-b922-bacef89a9084","Type":"ContainerDied","Data":"2ff7d7e46d59a3a6b86c4493157ace25c9bf517176b888ba2198cd7bfb887d91"} Oct 30 21:45:03.023728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.023714 4526 scope.go:122] "RemoveContainer" containerID="2ff7d7e46d59a3a6b86c4493157ace25c9bf517176b888ba2198cd7bfb887d91" Oct 30 21:45:03.025442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.025419 4526 generic.go:356] "Generic (PLEG): container finished" podID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerID="abce576da76c56783a086fda40f1621ed8e91e948f23b2c9910d361eebc4135e" exitCode=60 Oct 30 21:45:03.025602 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.025518 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" event={"ID":"cafeeafb-8b05-4378-bb7f-102784b2d91b","Type":"ContainerDied","Data":"abce576da76c56783a086fda40f1621ed8e91e948f23b2c9910d361eebc4135e"} Oct 30 21:45:03.025744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.025730 4526 scope.go:122] "RemoveContainer" containerID="abce576da76c56783a086fda40f1621ed8e91e948f23b2c9910d361eebc4135e" Oct 30 21:45:03.027720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.027699 4526 generic.go:356] "Generic (PLEG): container finished" podID="a7d44537-c273-47c3-b64e-b3225d200495" containerID="6d0319ddaf968187065bd8976693784fe4126dd89ec50658c2d420f03a163c9d" exitCode=60 Oct 30 21:45:03.027821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.027719 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" event={"ID":"a7d44537-c273-47c3-b64e-b3225d200495","Type":"ContainerDied","Data":"6d0319ddaf968187065bd8976693784fe4126dd89ec50658c2d420f03a163c9d"} Oct 30 21:45:03.028075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.028057 4526 scope.go:122] "RemoveContainer" containerID="6d0319ddaf968187065bd8976693784fe4126dd89ec50658c2d420f03a163c9d" Oct 30 21:45:03.031044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.031020 4526 generic.go:356] "Generic (PLEG): container finished" podID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerID="07764d81c360a1c71db48e057c9f7d730ce0886662a69772375b2cc04fbf0268" exitCode=60 Oct 30 21:45:03.031147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.031059 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" event={"ID":"1f63cc29-b531-4c4f-af3e-81a3704576ff","Type":"ContainerDied","Data":"07764d81c360a1c71db48e057c9f7d730ce0886662a69772375b2cc04fbf0268"} Oct 30 21:45:03.031699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:03.031677 4526 scope.go:122] "RemoveContainer" containerID="07764d81c360a1c71db48e057c9f7d730ce0886662a69772375b2cc04fbf0268" Oct 30 21:45:04.042831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.042790 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" event={"ID":"0cd16d13-ff85-4eb2-a567-2800b627aa54","Type":"ContainerStarted","Data":"6e124c39ebaa5dab991560462e2fb82697c56615b24246a07744cb4c837a399e"} Oct 30 21:45:04.044944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.044912 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" event={"ID":"acdef6a3-2a30-4bbe-ab10-64abc1239a66","Type":"ContainerStarted","Data":"7fe88a943989b2c820c4aefd5e326dd9965d61be9219cc3e92f73c578b9ecd5e"} Oct 30 21:45:04.046926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.046905 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" event={"ID":"17547078-dc7b-473d-ac5e-1cd327ffac61","Type":"ContainerStarted","Data":"44ca994641d0fc2448feacf536c8fce64dad950ad8253889904b684009f22168"} Oct 30 21:45:04.048806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.048784 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" event={"ID":"35eb2ffd-c29e-494a-8aa9-db939e290075","Type":"ContainerStarted","Data":"900c91fa0286b2bd2f2ffee4dc2cf50dc61858c0d1c92dd351892685e844088c"} Oct 30 21:45:04.050665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.050645 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" event={"ID":"a8d299b1-ab44-421a-b922-bacef89a9084","Type":"ContainerStarted","Data":"62b711dd09fd802ef63812b816c4df53974046f434b2b38235e888bb8e345529"} Oct 30 21:45:04.052544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.052518 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" event={"ID":"cafeeafb-8b05-4378-bb7f-102784b2d91b","Type":"ContainerStarted","Data":"27d3d18eeb8e1499a86f2bf8b57f5415a4f978bdc5a17196484de64952e6231e"} Oct 30 21:45:04.054469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.054446 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" event={"ID":"a7d44537-c273-47c3-b64e-b3225d200495","Type":"ContainerStarted","Data":"c3b527152f9c4ca81754b8f0846e96813229a735b34c9a3db03edcc77bcbb8f4"} Oct 30 21:45:04.056143 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.056124 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" event={"ID":"1f63cc29-b531-4c4f-af3e-81a3704576ff","Type":"ContainerStarted","Data":"3381711849b5871446c3dd59ce2a9840f6f78699e7c75913f96860bb52f04295"} Oct 30 21:45:04.387102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.387051 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" podStartSLOduration=4.387036977 podStartE2EDuration="4.387036977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.384904775 +0000 UTC m=+1694.518277974" watchObservedRunningTime="2026-10-30 21:45:04.387036977 +0000 UTC m=+1694.520410208" Oct 30 21:45:04.501291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.501242 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" podStartSLOduration=4.501228265 podStartE2EDuration="4.501228265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.496791166 +0000 UTC m=+1694.630164365" watchObservedRunningTime="2026-10-30 21:45:04.501228265 +0000 UTC m=+1694.634601523" Oct 30 21:45:04.612919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.612856 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" podStartSLOduration=4.612841102 podStartE2EDuration="4.612841102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.608366016 +0000 UTC m=+1694.741739215" watchObservedRunningTime="2026-10-30 21:45:04.612841102 +0000 UTC m=+1694.746214299" Oct 30 21:45:04.944954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:04.944899 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" podStartSLOduration=4.944884352 podStartE2EDuration="4.944884352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.944332346 +0000 UTC m=+1695.077705559" watchObservedRunningTime="2026-10-30 21:45:04.944884352 +0000 UTC m=+1695.078257550" Oct 30 21:45:05.068295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.068255 4526 generic.go:356] "Generic (PLEG): container finished" podID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerID="900c91fa0286b2bd2f2ffee4dc2cf50dc61858c0d1c92dd351892685e844088c" exitCode=60 Oct 30 21:45:05.068833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.068342 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" event={"ID":"35eb2ffd-c29e-494a-8aa9-db939e290075","Type":"ContainerDied","Data":"900c91fa0286b2bd2f2ffee4dc2cf50dc61858c0d1c92dd351892685e844088c"} Oct 30 21:45:05.068833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.068416 4526 scope.go:122] "RemoveContainer" containerID="7c78f3ecc67501299ed4662ac2ea6997ef61f82160f39299690b3bc668115286" Oct 30 21:45:05.068914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.068833 4526 scope.go:122] "RemoveContainer" containerID="900c91fa0286b2bd2f2ffee4dc2cf50dc61858c0d1c92dd351892685e844088c" Oct 30 21:45:05.069288 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:05.069260 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-22qvs_odh-ai-gateway-infra(35eb2ffd-c29e-494a-8aa9-db939e290075)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" Oct 30 21:45:05.070833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.070812 4526 generic.go:356] "Generic (PLEG): container finished" podID="a8d299b1-ab44-421a-b922-bacef89a9084" containerID="62b711dd09fd802ef63812b816c4df53974046f434b2b38235e888bb8e345529" exitCode=60 Oct 30 21:45:05.070946 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.070882 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" event={"ID":"a8d299b1-ab44-421a-b922-bacef89a9084","Type":"ContainerDied","Data":"62b711dd09fd802ef63812b816c4df53974046f434b2b38235e888bb8e345529"} Oct 30 21:45:05.071232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.071219 4526 scope.go:122] "RemoveContainer" containerID="62b711dd09fd802ef63812b816c4df53974046f434b2b38235e888bb8e345529" Oct 30 21:45:05.071685 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:05.071665 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs_odh-ai-gateway-infra(a8d299b1-ab44-421a-b922-bacef89a9084)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" Oct 30 21:45:05.073111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.073089 4526 generic.go:356] "Generic (PLEG): container finished" podID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerID="27d3d18eeb8e1499a86f2bf8b57f5415a4f978bdc5a17196484de64952e6231e" exitCode=60 Oct 30 21:45:05.073224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.073173 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" event={"ID":"cafeeafb-8b05-4378-bb7f-102784b2d91b","Type":"ContainerDied","Data":"27d3d18eeb8e1499a86f2bf8b57f5415a4f978bdc5a17196484de64952e6231e"} Oct 30 21:45:05.073460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.073444 4526 scope.go:122] "RemoveContainer" containerID="27d3d18eeb8e1499a86f2bf8b57f5415a4f978bdc5a17196484de64952e6231e" Oct 30 21:45:05.073862 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:05.073840 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v_odh-ai-gateway-infra(cafeeafb-8b05-4378-bb7f-102784b2d91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" Oct 30 21:45:05.075205 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.075188 4526 generic.go:356] "Generic (PLEG): container finished" podID="a7d44537-c273-47c3-b64e-b3225d200495" containerID="c3b527152f9c4ca81754b8f0846e96813229a735b34c9a3db03edcc77bcbb8f4" exitCode=60 Oct 30 21:45:05.075342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.075320 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" event={"ID":"a7d44537-c273-47c3-b64e-b3225d200495","Type":"ContainerDied","Data":"c3b527152f9c4ca81754b8f0846e96813229a735b34c9a3db03edcc77bcbb8f4"} Oct 30 21:45:05.075568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.075538 4526 scope.go:122] "RemoveContainer" containerID="c3b527152f9c4ca81754b8f0846e96813229a735b34c9a3db03edcc77bcbb8f4" Oct 30 21:45:05.075928 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:05.075904 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq_odh-ai-gateway-infra(a7d44537-c273-47c3-b64e-b3225d200495)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" podUID="a7d44537-c273-47c3-b64e-b3225d200495" Oct 30 21:45:05.077313 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.077286 4526 generic.go:356] "Generic (PLEG): container finished" podID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerID="3381711849b5871446c3dd59ce2a9840f6f78699e7c75913f96860bb52f04295" exitCode=60 Oct 30 21:45:05.077408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.077376 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" event={"ID":"1f63cc29-b531-4c4f-af3e-81a3704576ff","Type":"ContainerDied","Data":"3381711849b5871446c3dd59ce2a9840f6f78699e7c75913f96860bb52f04295"} Oct 30 21:45:05.077639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.077622 4526 scope.go:122] "RemoveContainer" containerID="3381711849b5871446c3dd59ce2a9840f6f78699e7c75913f96860bb52f04295" Oct 30 21:45:05.077964 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:05.077944 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v_odh-ai-gateway-infra(1f63cc29-b531-4c4f-af3e-81a3704576ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" Oct 30 21:45:05.079411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.079388 4526 generic.go:356] "Generic (PLEG): container finished" podID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerID="6e124c39ebaa5dab991560462e2fb82697c56615b24246a07744cb4c837a399e" exitCode=60 Oct 30 21:45:05.079524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.079470 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" event={"ID":"0cd16d13-ff85-4eb2-a567-2800b627aa54","Type":"ContainerDied","Data":"6e124c39ebaa5dab991560462e2fb82697c56615b24246a07744cb4c837a399e"} Oct 30 21:45:05.079777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.079756 4526 scope.go:122] "RemoveContainer" containerID="6e124c39ebaa5dab991560462e2fb82697c56615b24246a07744cb4c837a399e" Oct 30 21:45:05.080037 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:05.080012 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr_odh-ai-gateway-infra(0cd16d13-ff85-4eb2-a567-2800b627aa54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" Oct 30 21:45:05.081668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.081650 4526 generic.go:356] "Generic (PLEG): container finished" podID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerID="7fe88a943989b2c820c4aefd5e326dd9965d61be9219cc3e92f73c578b9ecd5e" exitCode=60 Oct 30 21:45:05.081753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.081731 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" event={"ID":"acdef6a3-2a30-4bbe-ab10-64abc1239a66","Type":"ContainerDied","Data":"7fe88a943989b2c820c4aefd5e326dd9965d61be9219cc3e92f73c578b9ecd5e"} Oct 30 21:45:05.081966 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.081947 4526 scope.go:122] "RemoveContainer" containerID="7fe88a943989b2c820c4aefd5e326dd9965d61be9219cc3e92f73c578b9ecd5e" Oct 30 21:45:05.082266 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:05.082244 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv_odh-ai-gateway-infra(acdef6a3-2a30-4bbe-ab10-64abc1239a66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" Oct 30 21:45:05.083792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.083772 4526 generic.go:356] "Generic (PLEG): container finished" podID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerID="44ca994641d0fc2448feacf536c8fce64dad950ad8253889904b684009f22168" exitCode=60 Oct 30 21:45:05.083895 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.083821 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" event={"ID":"17547078-dc7b-473d-ac5e-1cd327ffac61","Type":"ContainerDied","Data":"44ca994641d0fc2448feacf536c8fce64dad950ad8253889904b684009f22168"} Oct 30 21:45:05.084001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.083987 4526 scope.go:122] "RemoveContainer" containerID="44ca994641d0fc2448feacf536c8fce64dad950ad8253889904b684009f22168" Oct 30 21:45:05.084225 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:05.084209 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h_odh-ai-gateway-infra(17547078-dc7b-473d-ac5e-1cd327ffac61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" Oct 30 21:45:05.085515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.085499 4526 scope.go:122] "RemoveContainer" containerID="2ff7d7e46d59a3a6b86c4493157ace25c9bf517176b888ba2198cd7bfb887d91" Oct 30 21:45:05.100105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.100087 4526 scope.go:122] "RemoveContainer" containerID="abce576da76c56783a086fda40f1621ed8e91e948f23b2c9910d361eebc4135e" Oct 30 21:45:05.115372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.115338 4526 scope.go:122] "RemoveContainer" containerID="6d0319ddaf968187065bd8976693784fe4126dd89ec50658c2d420f03a163c9d" Oct 30 21:45:05.164895 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.149542 4526 scope.go:122] "RemoveContainer" containerID="07764d81c360a1c71db48e057c9f7d730ce0886662a69772375b2cc04fbf0268" Oct 30 21:45:05.200754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.200725 4526 scope.go:122] "RemoveContainer" containerID="6b75721fa3d4a811bffe563ff6d46c276541e83028f3e47bb53e497d3ea6fef2" Oct 30 21:45:05.282304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.282276 4526 scope.go:122] "RemoveContainer" containerID="9dd18c51d3eb396c31f775ef955d0d1de73f2847892aada36353cbabc76e42e5" Oct 30 21:45:05.305968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:05.305942 4526 scope.go:122] "RemoveContainer" containerID="30fefa1a603d2e2858b4d8d1740e0e14b2758ae63af6e1357b96b71f0f6ecba5" Oct 30 21:45:06.096045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:06.096017 4526 scope.go:122] "RemoveContainer" containerID="900c91fa0286b2bd2f2ffee4dc2cf50dc61858c0d1c92dd351892685e844088c" Oct 30 21:45:06.096522 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:06.096502 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-22qvs_odh-ai-gateway-infra(35eb2ffd-c29e-494a-8aa9-db939e290075)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" Oct 30 21:45:06.098124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:06.098067 4526 scope.go:122] "RemoveContainer" containerID="62b711dd09fd802ef63812b816c4df53974046f434b2b38235e888bb8e345529" Oct 30 21:45:06.098439 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:06.098417 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs_odh-ai-gateway-infra(a8d299b1-ab44-421a-b922-bacef89a9084)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" Oct 30 21:45:06.099561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:06.099539 4526 scope.go:122] "RemoveContainer" containerID="27d3d18eeb8e1499a86f2bf8b57f5415a4f978bdc5a17196484de64952e6231e" Oct 30 21:45:06.099885 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:06.099860 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v_odh-ai-gateway-infra(cafeeafb-8b05-4378-bb7f-102784b2d91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" Oct 30 21:45:06.101247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:06.101225 4526 scope.go:122] "RemoveContainer" containerID="c3b527152f9c4ca81754b8f0846e96813229a735b34c9a3db03edcc77bcbb8f4" Oct 30 21:45:06.101548 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:06.101529 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq_odh-ai-gateway-infra(a7d44537-c273-47c3-b64e-b3225d200495)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" podUID="a7d44537-c273-47c3-b64e-b3225d200495" Oct 30 21:45:06.103245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:06.103228 4526 scope.go:122] "RemoveContainer" containerID="3381711849b5871446c3dd59ce2a9840f6f78699e7c75913f96860bb52f04295" Oct 30 21:45:06.103552 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:06.103532 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v_odh-ai-gateway-infra(1f63cc29-b531-4c4f-af3e-81a3704576ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" Oct 30 21:45:06.105089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:06.105052 4526 scope.go:122] "RemoveContainer" containerID="6e124c39ebaa5dab991560462e2fb82697c56615b24246a07744cb4c837a399e" Oct 30 21:45:06.105376 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:06.105348 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr_odh-ai-gateway-infra(0cd16d13-ff85-4eb2-a567-2800b627aa54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" Oct 30 21:45:06.107546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:06.107465 4526 scope.go:122] "RemoveContainer" containerID="7fe88a943989b2c820c4aefd5e326dd9965d61be9219cc3e92f73c578b9ecd5e" Oct 30 21:45:06.107863 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:06.107843 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv_odh-ai-gateway-infra(acdef6a3-2a30-4bbe-ab10-64abc1239a66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" Oct 30 21:45:06.109709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:06.109674 4526 scope.go:122] "RemoveContainer" containerID="44ca994641d0fc2448feacf536c8fce64dad950ad8253889904b684009f22168" Oct 30 21:45:06.110019 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:06.109995 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h_odh-ai-gateway-infra(17547078-dc7b-473d-ac5e-1cd327ffac61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" Oct 30 21:45:08.313854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.313820 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr"] Oct 30 21:45:08.320424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.320378 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.376152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.376123 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr"] Oct 30 21:45:08.418423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.418378 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:45:08.462991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.462952 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.463174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.463021 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqznv\" (UniqueName: \"kubernetes.io/projected/6aa12cd0-2d28-4dc9-988f-687b912375e5-kube-api-access-jqznv\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.463269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.463243 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.565008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.564909 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.565008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.564998 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqznv\" (UniqueName: \"kubernetes.io/projected/6aa12cd0-2d28-4dc9-988f-687b912375e5-kube-api-access-jqznv\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.565248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.565210 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.565637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.565616 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.567772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.567742 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.622546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.622512 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqznv\" (UniqueName: \"kubernetes.io/projected/6aa12cd0-2d28-4dc9-988f-687b912375e5-kube-api-access-jqznv\") pod \"maas-api-revoke-keys-e2e-ait-1915737c-vhtzr\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:08.636623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:08.636593 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:09.056197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:09.047286 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr"] Oct 30 21:45:09.139420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:09.139378 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" event={"ID":"6aa12cd0-2d28-4dc9-988f-687b912375e5","Type":"ContainerStarted","Data":"80ef7046163f7549d18f11c205e1fa6f26c419b9d4fcefa908004a6db03ec61e"} Oct 30 21:45:10.150584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:10.150539 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" event={"ID":"6aa12cd0-2d28-4dc9-988f-687b912375e5","Type":"ContainerStarted","Data":"5f18bc40369200ea59730b206d1f9b1a0268d29dbf540ce089e53fc4c9935c25"} Oct 30 21:45:10.268805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:10.268749 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" podStartSLOduration=2.26873202 podStartE2EDuration="2.26873202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:10.262872942 +0000 UTC m=+1700.396246150" watchObservedRunningTime="2026-10-30 21:45:10.26873202 +0000 UTC m=+1700.402105228" Oct 30 21:45:11.049957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:11.049922 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn"] Oct 30 21:45:11.099350 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:11.099309 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db12a902-lzbcn"] Oct 30 21:45:11.161858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:11.161824 4526 generic.go:356] "Generic (PLEG): container finished" podID="6aa12cd0-2d28-4dc9-988f-687b912375e5" containerID="5f18bc40369200ea59730b206d1f9b1a0268d29dbf540ce089e53fc4c9935c25" exitCode=0 Oct 30 21:45:11.162327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:11.161897 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" event={"ID":"6aa12cd0-2d28-4dc9-988f-687b912375e5","Type":"ContainerDied","Data":"5f18bc40369200ea59730b206d1f9b1a0268d29dbf540ce089e53fc4c9935c25"} Oct 30 21:45:11.989248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:11.989211 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378c3fe9-9948-4a53-9729-92cdd552b886" path="/var/lib/kubelet/pods/378c3fe9-9948-4a53-9729-92cdd552b886/volumes" Oct 30 21:45:12.344802 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.344775 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:12.432646 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.432610 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-openshift-ca\") pod \"6aa12cd0-2d28-4dc9-988f-687b912375e5\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " Oct 30 21:45:12.432814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.432785 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa12cd0-2d28-4dc9-988f-687b912375e5-kube-api-access-jqznv\" (UniqueName: \"kubernetes.io/projected/6aa12cd0-2d28-4dc9-988f-687b912375e5-kube-api-access-jqznv\") pod \"6aa12cd0-2d28-4dc9-988f-687b912375e5\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " Oct 30 21:45:12.432857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.432849 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-xks-ca\") pod \"6aa12cd0-2d28-4dc9-988f-687b912375e5\" (UID: \"6aa12cd0-2d28-4dc9-988f-687b912375e5\") " Oct 30 21:45:12.432974 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.432952 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-openshift-ca" pod "6aa12cd0-2d28-4dc9-988f-687b912375e5" (UID: "6aa12cd0-2d28-4dc9-988f-687b912375e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:12.433549 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.433528 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:12.435028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.435010 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa12cd0-2d28-4dc9-988f-687b912375e5-kube-api-access-jqznv" pod "6aa12cd0-2d28-4dc9-988f-687b912375e5" (UID: "6aa12cd0-2d28-4dc9-988f-687b912375e5"). InnerVolumeSpecName "kube-api-access-jqznv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:12.435159 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.435145 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-xks-ca" pod "6aa12cd0-2d28-4dc9-988f-687b912375e5" (UID: "6aa12cd0-2d28-4dc9-988f-687b912375e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:12.486310 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:12.486274 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:45:12.534843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.534807 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqznv\" (UniqueName: \"kubernetes.io/projected/6aa12cd0-2d28-4dc9-988f-687b912375e5-kube-api-access-jqznv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:12.534843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:12.534839 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa12cd0-2d28-4dc9-988f-687b912375e5-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:13.049253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:13.049226 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql"] Oct 30 21:45:13.053812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:13.053786 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e180a309-4bhql"] Oct 30 21:45:13.180657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:13.180624 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" Oct 30 21:45:13.180840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:13.180664 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr" event={"ID":"6aa12cd0-2d28-4dc9-988f-687b912375e5","Type":"ContainerDied","Data":"80ef7046163f7549d18f11c205e1fa6f26c419b9d4fcefa908004a6db03ec61e"} Oct 30 21:45:13.180840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:13.180711 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ef7046163f7549d18f11c205e1fa6f26c419b9d4fcefa908004a6db03ec61e" Oct 30 21:45:13.989335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:13.989301 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84471d9-a81d-48f3-aed7-91bdd7bca242" path="/var/lib/kubelet/pods/b84471d9-a81d-48f3-aed7-91bdd7bca242/volumes" Oct 30 21:45:14.978474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:14.978445 4526 scope.go:122] "RemoveContainer" containerID="44ca994641d0fc2448feacf536c8fce64dad950ad8253889904b684009f22168" Oct 30 21:45:15.980645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:15.980604 4526 scope.go:122] "RemoveContainer" containerID="3381711849b5871446c3dd59ce2a9840f6f78699e7c75913f96860bb52f04295" Oct 30 21:45:15.983407 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:15.982127 4526 scope.go:122] "RemoveContainer" containerID="62b711dd09fd802ef63812b816c4df53974046f434b2b38235e888bb8e345529" Oct 30 21:45:15.983407 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:15.982434 4526 scope.go:122] "RemoveContainer" containerID="27d3d18eeb8e1499a86f2bf8b57f5415a4f978bdc5a17196484de64952e6231e" Oct 30 21:45:16.215371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:16.214319 4526 generic.go:356] "Generic (PLEG): container finished" podID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerID="9de4d1e267a343d429f497dafa4b61295b3e04c089227355e1a8e08fffe9e253" exitCode=60 Oct 30 21:45:16.215371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:16.214354 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" event={"ID":"17547078-dc7b-473d-ac5e-1cd327ffac61","Type":"ContainerDied","Data":"9de4d1e267a343d429f497dafa4b61295b3e04c089227355e1a8e08fffe9e253"} Oct 30 21:45:16.215371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:16.214442 4526 scope.go:122] "RemoveContainer" containerID="44ca994641d0fc2448feacf536c8fce64dad950ad8253889904b684009f22168" Oct 30 21:45:16.215371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:16.215046 4526 scope.go:122] "RemoveContainer" containerID="9de4d1e267a343d429f497dafa4b61295b3e04c089227355e1a8e08fffe9e253" Oct 30 21:45:16.215752 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:16.215716 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h_odh-ai-gateway-infra(17547078-dc7b-473d-ac5e-1cd327ffac61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" Oct 30 21:45:16.978543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:16.978509 4526 scope.go:122] "RemoveContainer" containerID="900c91fa0286b2bd2f2ffee4dc2cf50dc61858c0d1c92dd351892685e844088c" Oct 30 21:45:16.978752 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:16.978732 4526 scope.go:122] "RemoveContainer" containerID="c3b527152f9c4ca81754b8f0846e96813229a735b34c9a3db03edcc77bcbb8f4" Oct 30 21:45:17.224892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.224865 4526 generic.go:356] "Generic (PLEG): container finished" podID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerID="a0baf6f92b6f30bd94be7cc8f813c504b6dbd289b124295e4f54b554caa0d493" exitCode=60 Oct 30 21:45:17.225318 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.224941 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" event={"ID":"1f63cc29-b531-4c4f-af3e-81a3704576ff","Type":"ContainerDied","Data":"a0baf6f92b6f30bd94be7cc8f813c504b6dbd289b124295e4f54b554caa0d493"} Oct 30 21:45:17.225318 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.225007 4526 scope.go:122] "RemoveContainer" containerID="3381711849b5871446c3dd59ce2a9840f6f78699e7c75913f96860bb52f04295" Oct 30 21:45:17.225533 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.225481 4526 scope.go:122] "RemoveContainer" containerID="a0baf6f92b6f30bd94be7cc8f813c504b6dbd289b124295e4f54b554caa0d493" Oct 30 21:45:17.225957 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:17.225937 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v_odh-ai-gateway-infra(1f63cc29-b531-4c4f-af3e-81a3704576ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" Oct 30 21:45:17.229285 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.229228 4526 generic.go:356] "Generic (PLEG): container finished" podID="a8d299b1-ab44-421a-b922-bacef89a9084" containerID="7328cfca677200dab8a8c3e81b1b592ded394a77e8c7669c9973c742ba14c77b" exitCode=60 Oct 30 21:45:17.229394 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.229303 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" event={"ID":"a8d299b1-ab44-421a-b922-bacef89a9084","Type":"ContainerDied","Data":"7328cfca677200dab8a8c3e81b1b592ded394a77e8c7669c9973c742ba14c77b"} Oct 30 21:45:17.229765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.229750 4526 scope.go:122] "RemoveContainer" containerID="7328cfca677200dab8a8c3e81b1b592ded394a77e8c7669c9973c742ba14c77b" Oct 30 21:45:17.230142 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:17.230121 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs_odh-ai-gateway-infra(a8d299b1-ab44-421a-b922-bacef89a9084)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" Oct 30 21:45:17.231448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.231428 4526 generic.go:356] "Generic (PLEG): container finished" podID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerID="446a1c43e2a8485a9746e4d9a23033e00d745d4d0dbf4fa9e82f124667374f61" exitCode=60 Oct 30 21:45:17.231611 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.231523 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" event={"ID":"cafeeafb-8b05-4378-bb7f-102784b2d91b","Type":"ContainerDied","Data":"446a1c43e2a8485a9746e4d9a23033e00d745d4d0dbf4fa9e82f124667374f61"} Oct 30 21:45:17.231897 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.231884 4526 scope.go:122] "RemoveContainer" containerID="446a1c43e2a8485a9746e4d9a23033e00d745d4d0dbf4fa9e82f124667374f61" Oct 30 21:45:17.232188 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:17.232171 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v_odh-ai-gateway-infra(cafeeafb-8b05-4378-bb7f-102784b2d91b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" Oct 30 21:45:17.270383 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.270363 4526 scope.go:122] "RemoveContainer" containerID="62b711dd09fd802ef63812b816c4df53974046f434b2b38235e888bb8e345529" Oct 30 21:45:17.292798 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.292762 4526 scope.go:122] "RemoveContainer" containerID="27d3d18eeb8e1499a86f2bf8b57f5415a4f978bdc5a17196484de64952e6231e" Oct 30 21:45:17.339554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.339508 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h"] Oct 30 21:45:17.586866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.586842 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:17.724595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.724569 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17547078-dc7b-473d-ac5e-1cd327ffac61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17547078-dc7b-473d-ac5e-1cd327ffac61-maas-api-service-ca\") pod \"17547078-dc7b-473d-ac5e-1cd327ffac61\" (UID: \"17547078-dc7b-473d-ac5e-1cd327ffac61\") " Oct 30 21:45:17.724767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.724620 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17547078-dc7b-473d-ac5e-1cd327ffac61-kube-api-access-7zwfh\" (UniqueName: \"kubernetes.io/projected/17547078-dc7b-473d-ac5e-1cd327ffac61-kube-api-access-7zwfh\") pod \"17547078-dc7b-473d-ac5e-1cd327ffac61\" (UID: \"17547078-dc7b-473d-ac5e-1cd327ffac61\") " Oct 30 21:45:17.724970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.724945 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17547078-dc7b-473d-ac5e-1cd327ffac61-maas-api-service-ca" pod "17547078-dc7b-473d-ac5e-1cd327ffac61" (UID: "17547078-dc7b-473d-ac5e-1cd327ffac61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:17.725334 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.725316 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17547078-dc7b-473d-ac5e-1cd327ffac61-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.726774 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.726754 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17547078-dc7b-473d-ac5e-1cd327ffac61-kube-api-access-7zwfh" pod "17547078-dc7b-473d-ac5e-1cd327ffac61" (UID: "17547078-dc7b-473d-ac5e-1cd327ffac61"). InnerVolumeSpecName "kube-api-access-7zwfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:17.830507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.827187 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zwfh\" (UniqueName: \"kubernetes.io/projected/17547078-dc7b-473d-ac5e-1cd327ffac61-kube-api-access-7zwfh\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:17.978217 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.978183 4526 scope.go:122] "RemoveContainer" containerID="6e124c39ebaa5dab991560462e2fb82697c56615b24246a07744cb4c837a399e" Oct 30 21:45:17.978365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:17.978346 4526 scope.go:122] "RemoveContainer" containerID="7fe88a943989b2c820c4aefd5e326dd9965d61be9219cc3e92f73c578b9ecd5e" Oct 30 21:45:18.242705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.242677 4526 generic.go:356] "Generic (PLEG): container finished" podID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerID="b174c1e7db62aa79fdf3353ca06a564f4a277f105264589797738661688377de" exitCode=60 Oct 30 21:45:18.243105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.242726 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" event={"ID":"35eb2ffd-c29e-494a-8aa9-db939e290075","Type":"ContainerDied","Data":"b174c1e7db62aa79fdf3353ca06a564f4a277f105264589797738661688377de"} Oct 30 21:45:18.243105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.242787 4526 scope.go:122] "RemoveContainer" containerID="900c91fa0286b2bd2f2ffee4dc2cf50dc61858c0d1c92dd351892685e844088c" Oct 30 21:45:18.243317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.243295 4526 scope.go:122] "RemoveContainer" containerID="b174c1e7db62aa79fdf3353ca06a564f4a277f105264589797738661688377de" Oct 30 21:45:18.243783 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:18.243760 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-22qvs_odh-ai-gateway-infra(35eb2ffd-c29e-494a-8aa9-db939e290075)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" Oct 30 21:45:18.251424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.250524 4526 generic.go:356] "Generic (PLEG): container finished" podID="a7d44537-c273-47c3-b64e-b3225d200495" containerID="71a30a9ba205787f35cd2d7726798931c82c0a57ebf75f8abd378cfc89970901" exitCode=60 Oct 30 21:45:18.251424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.250707 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" event={"ID":"a7d44537-c273-47c3-b64e-b3225d200495","Type":"ContainerDied","Data":"71a30a9ba205787f35cd2d7726798931c82c0a57ebf75f8abd378cfc89970901"} Oct 30 21:45:18.251666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.251615 4526 scope.go:122] "RemoveContainer" containerID="71a30a9ba205787f35cd2d7726798931c82c0a57ebf75f8abd378cfc89970901" Oct 30 21:45:18.252076 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:18.252051 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq_odh-ai-gateway-infra(a7d44537-c273-47c3-b64e-b3225d200495)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" podUID="a7d44537-c273-47c3-b64e-b3225d200495" Oct 30 21:45:18.255944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.255918 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" event={"ID":"17547078-dc7b-473d-ac5e-1cd327ffac61","Type":"ContainerDied","Data":"de7e1980ccf7e33a3d6779e2291646d2776ce5be16336119d1e36c3499f5a044"} Oct 30 21:45:18.256043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.256017 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h" Oct 30 21:45:18.319452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.319432 4526 scope.go:122] "RemoveContainer" containerID="c3b527152f9c4ca81754b8f0846e96813229a735b34c9a3db03edcc77bcbb8f4" Oct 30 21:45:18.337366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.337334 4526 scope.go:122] "RemoveContainer" containerID="9de4d1e267a343d429f497dafa4b61295b3e04c089227355e1a8e08fffe9e253" Oct 30 21:45:18.370934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.370903 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v"] Oct 30 21:45:18.591327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.591296 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h"] Oct 30 21:45:18.596593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.596564 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-446ca2-29889945-2842h"] Oct 30 21:45:18.601357 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.601332 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v"] Oct 30 21:45:18.629783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.629758 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:18.715626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.710942 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs"] Oct 30 21:45:18.743696 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.743663 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f63cc29-b531-4c4f-af3e-81a3704576ff-kube-api-access-rlqkj\" (UniqueName: \"kubernetes.io/projected/1f63cc29-b531-4c4f-af3e-81a3704576ff-kube-api-access-rlqkj\") pod \"1f63cc29-b531-4c4f-af3e-81a3704576ff\" (UID: \"1f63cc29-b531-4c4f-af3e-81a3704576ff\") " Oct 30 21:45:18.743896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.743860 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f63cc29-b531-4c4f-af3e-81a3704576ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f63cc29-b531-4c4f-af3e-81a3704576ff-maas-api-service-ca\") pod \"1f63cc29-b531-4c4f-af3e-81a3704576ff\" (UID: \"1f63cc29-b531-4c4f-af3e-81a3704576ff\") " Oct 30 21:45:18.744651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.744607 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f63cc29-b531-4c4f-af3e-81a3704576ff-maas-api-service-ca" pod "1f63cc29-b531-4c4f-af3e-81a3704576ff" (UID: "1f63cc29-b531-4c4f-af3e-81a3704576ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:18.747847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.747227 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f63cc29-b531-4c4f-af3e-81a3704576ff-kube-api-access-rlqkj" pod "1f63cc29-b531-4c4f-af3e-81a3704576ff" (UID: "1f63cc29-b531-4c4f-af3e-81a3704576ff"). InnerVolumeSpecName "kube-api-access-rlqkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:18.779345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.779309 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:18.849830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.849792 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f63cc29-b531-4c4f-af3e-81a3704576ff-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:18.849830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.849831 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqkj\" (UniqueName: \"kubernetes.io/projected/1f63cc29-b531-4c4f-af3e-81a3704576ff-kube-api-access-rlqkj\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:18.883182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.883154 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:18.950669 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.950638 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafeeafb-8b05-4378-bb7f-102784b2d91b-kube-api-access-72h2q\" (UniqueName: \"kubernetes.io/projected/cafeeafb-8b05-4378-bb7f-102784b2d91b-kube-api-access-72h2q\") pod \"cafeeafb-8b05-4378-bb7f-102784b2d91b\" (UID: \"cafeeafb-8b05-4378-bb7f-102784b2d91b\") " Oct 30 21:45:18.950862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.950697 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafeeafb-8b05-4378-bb7f-102784b2d91b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafeeafb-8b05-4378-bb7f-102784b2d91b-maas-api-service-ca\") pod \"cafeeafb-8b05-4378-bb7f-102784b2d91b\" (UID: \"cafeeafb-8b05-4378-bb7f-102784b2d91b\") " Oct 30 21:45:18.951074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.951051 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafeeafb-8b05-4378-bb7f-102784b2d91b-maas-api-service-ca" pod "cafeeafb-8b05-4378-bb7f-102784b2d91b" (UID: "cafeeafb-8b05-4378-bb7f-102784b2d91b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:18.951433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.951418 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cafeeafb-8b05-4378-bb7f-102784b2d91b-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:18.953028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:18.953009 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafeeafb-8b05-4378-bb7f-102784b2d91b-kube-api-access-72h2q" pod "cafeeafb-8b05-4378-bb7f-102784b2d91b" (UID: "cafeeafb-8b05-4378-bb7f-102784b2d91b"). InnerVolumeSpecName "kube-api-access-72h2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.052800 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.052765 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d299b1-ab44-421a-b922-bacef89a9084-kube-api-access-z7sc4\" (UniqueName: \"kubernetes.io/projected/a8d299b1-ab44-421a-b922-bacef89a9084-kube-api-access-z7sc4\") pod \"a8d299b1-ab44-421a-b922-bacef89a9084\" (UID: \"a8d299b1-ab44-421a-b922-bacef89a9084\") " Oct 30 21:45:19.052971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.052848 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d299b1-ab44-421a-b922-bacef89a9084-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d299b1-ab44-421a-b922-bacef89a9084-maas-api-service-ca\") pod \"a8d299b1-ab44-421a-b922-bacef89a9084\" (UID: \"a8d299b1-ab44-421a-b922-bacef89a9084\") " Oct 30 21:45:19.053206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.053182 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d299b1-ab44-421a-b922-bacef89a9084-maas-api-service-ca" pod "a8d299b1-ab44-421a-b922-bacef89a9084" (UID: "a8d299b1-ab44-421a-b922-bacef89a9084"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.053592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.053577 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72h2q\" (UniqueName: \"kubernetes.io/projected/cafeeafb-8b05-4378-bb7f-102784b2d91b-kube-api-access-72h2q\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.053646 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.053596 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d299b1-ab44-421a-b922-bacef89a9084-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.095386 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.095085 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d299b1-ab44-421a-b922-bacef89a9084-kube-api-access-z7sc4" pod "a8d299b1-ab44-421a-b922-bacef89a9084" (UID: "a8d299b1-ab44-421a-b922-bacef89a9084"). InnerVolumeSpecName "kube-api-access-z7sc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.155784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.155749 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7sc4\" (UniqueName: \"kubernetes.io/projected/a8d299b1-ab44-421a-b922-bacef89a9084-kube-api-access-z7sc4\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.272452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.272412 4526 generic.go:356] "Generic (PLEG): container finished" podID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerID="4d871a5c42bf42dcf7d5127bc4d1381f74599fd8a4b7a50f79107bd1063f291c" exitCode=60 Oct 30 21:45:19.272978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.272455 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" event={"ID":"acdef6a3-2a30-4bbe-ab10-64abc1239a66","Type":"ContainerDied","Data":"4d871a5c42bf42dcf7d5127bc4d1381f74599fd8a4b7a50f79107bd1063f291c"} Oct 30 21:45:19.272978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.272553 4526 scope.go:122] "RemoveContainer" containerID="7fe88a943989b2c820c4aefd5e326dd9965d61be9219cc3e92f73c578b9ecd5e" Oct 30 21:45:19.273138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.273120 4526 scope.go:122] "RemoveContainer" containerID="4d871a5c42bf42dcf7d5127bc4d1381f74599fd8a4b7a50f79107bd1063f291c" Oct 30 21:45:19.273649 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:19.273622 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv_odh-ai-gateway-infra(acdef6a3-2a30-4bbe-ab10-64abc1239a66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" Oct 30 21:45:19.279279 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.279252 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" Oct 30 21:45:19.279389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.279297 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs" event={"ID":"a8d299b1-ab44-421a-b922-bacef89a9084","Type":"ContainerDied","Data":"d3cfed769a6f1256b6d68ea3a33b46514d0f85a6dc5aab5fec5cc77d81f53797"} Oct 30 21:45:19.281268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.281245 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" event={"ID":"cafeeafb-8b05-4378-bb7f-102784b2d91b","Type":"ContainerDied","Data":"98b1b0764f92ee1bf6361dd9468d5a577f8e8aafca2fb2033676b2fc1fd93d39"} Oct 30 21:45:19.281372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.281328 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v" Oct 30 21:45:19.291301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.291281 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" Oct 30 21:45:19.291301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.291291 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v" event={"ID":"1f63cc29-b531-4c4f-af3e-81a3704576ff","Type":"ContainerDied","Data":"0486dd21e514ee6d452808c08ee40bd25a995363dff33fe4c44ff9a949ae423a"} Oct 30 21:45:19.291471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.291289 4526 scope.go:122] "RemoveContainer" containerID="7328cfca677200dab8a8c3e81b1b592ded394a77e8c7669c9973c742ba14c77b" Oct 30 21:45:19.293984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.293965 4526 generic.go:356] "Generic (PLEG): container finished" podID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerID="657e38d9f7c64016368ae71d564345b511469d4cb5772b48d0a31439abbd98f8" exitCode=60 Oct 30 21:45:19.294100 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.293990 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" event={"ID":"0cd16d13-ff85-4eb2-a567-2800b627aa54","Type":"ContainerDied","Data":"657e38d9f7c64016368ae71d564345b511469d4cb5772b48d0a31439abbd98f8"} Oct 30 21:45:19.294387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.294372 4526 scope.go:122] "RemoveContainer" containerID="657e38d9f7c64016368ae71d564345b511469d4cb5772b48d0a31439abbd98f8" Oct 30 21:45:19.294721 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:19.294689 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr_odh-ai-gateway-infra(0cd16d13-ff85-4eb2-a567-2800b627aa54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" Oct 30 21:45:19.303790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.303774 4526 scope.go:122] "RemoveContainer" containerID="446a1c43e2a8485a9746e4d9a23033e00d745d4d0dbf4fa9e82f124667374f61" Oct 30 21:45:19.314222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.314202 4526 scope.go:122] "RemoveContainer" containerID="a0baf6f92b6f30bd94be7cc8f813c504b6dbd289b124295e4f54b554caa0d493" Oct 30 21:45:19.325301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.325280 4526 scope.go:122] "RemoveContainer" containerID="6e124c39ebaa5dab991560462e2fb82697c56615b24246a07744cb4c837a399e" Oct 30 21:45:19.376822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.376794 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs"] Oct 30 21:45:19.490549 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.490174 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq"] Oct 30 21:45:19.563301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.563222 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:19.618273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.618229 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v"] Oct 30 21:45:19.626103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.626065 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e4dadd-29889945-mcw7v"] Oct 30 21:45:19.664662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.664622 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eb2ffd-c29e-494a-8aa9-db939e290075-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35eb2ffd-c29e-494a-8aa9-db939e290075-maas-api-service-ca\") pod \"35eb2ffd-c29e-494a-8aa9-db939e290075\" (UID: \"35eb2ffd-c29e-494a-8aa9-db939e290075\") " Oct 30 21:45:19.664994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.664973 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eb2ffd-c29e-494a-8aa9-db939e290075-maas-api-service-ca" pod "35eb2ffd-c29e-494a-8aa9-db939e290075" (UID: "35eb2ffd-c29e-494a-8aa9-db939e290075"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.665041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.665015 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eb2ffd-c29e-494a-8aa9-db939e290075-kube-api-access-mmztx\" (UniqueName: \"kubernetes.io/projected/35eb2ffd-c29e-494a-8aa9-db939e290075-kube-api-access-mmztx\") pod \"35eb2ffd-c29e-494a-8aa9-db939e290075\" (UID: \"35eb2ffd-c29e-494a-8aa9-db939e290075\") " Oct 30 21:45:19.665782 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.665758 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35eb2ffd-c29e-494a-8aa9-db939e290075-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.667316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.667291 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eb2ffd-c29e-494a-8aa9-db939e290075-kube-api-access-mmztx" pod "35eb2ffd-c29e-494a-8aa9-db939e290075" (UID: "35eb2ffd-c29e-494a-8aa9-db939e290075"). InnerVolumeSpecName "kube-api-access-mmztx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.703464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.703436 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:19.730562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.730528 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v"] Oct 30 21:45:19.735300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.735273 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1915737c-29889945-srz9v"] Oct 30 21:45:19.767133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.767104 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmztx\" (UniqueName: \"kubernetes.io/projected/35eb2ffd-c29e-494a-8aa9-db939e290075-kube-api-access-mmztx\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.841438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.841361 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs"] Oct 30 21:45:19.847900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.847876 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2ace1d-29889945-lvtxs"] Oct 30 21:45:19.868321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.868297 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d44537-c273-47c3-b64e-b3225d200495-kube-api-access-jhkvq\" (UniqueName: \"kubernetes.io/projected/a7d44537-c273-47c3-b64e-b3225d200495-kube-api-access-jhkvq\") pod \"a7d44537-c273-47c3-b64e-b3225d200495\" (UID: \"a7d44537-c273-47c3-b64e-b3225d200495\") " Oct 30 21:45:19.868675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.868635 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d44537-c273-47c3-b64e-b3225d200495-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d44537-c273-47c3-b64e-b3225d200495-maas-api-service-ca\") pod \"a7d44537-c273-47c3-b64e-b3225d200495\" (UID: \"a7d44537-c273-47c3-b64e-b3225d200495\") " Oct 30 21:45:19.868987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.868962 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d44537-c273-47c3-b64e-b3225d200495-maas-api-service-ca" pod "a7d44537-c273-47c3-b64e-b3225d200495" (UID: "a7d44537-c273-47c3-b64e-b3225d200495"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.869396 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.869376 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d44537-c273-47c3-b64e-b3225d200495-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.870699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.870676 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d44537-c273-47c3-b64e-b3225d200495-kube-api-access-jhkvq" pod "a7d44537-c273-47c3-b64e-b3225d200495" (UID: "a7d44537-c273-47c3-b64e-b3225d200495"). InnerVolumeSpecName "kube-api-access-jhkvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.876401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.876378 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz"] Oct 30 21:45:19.970773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.970738 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhkvq\" (UniqueName: \"kubernetes.io/projected/a7d44537-c273-47c3-b64e-b3225d200495-kube-api-access-jhkvq\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:19.989671 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.989638 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" path="/var/lib/kubelet/pods/17547078-dc7b-473d-ac5e-1cd327ffac61/volumes" Oct 30 21:45:19.990462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.990437 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" path="/var/lib/kubelet/pods/1f63cc29-b531-4c4f-af3e-81a3704576ff/volumes" Oct 30 21:45:19.991424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.991399 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" path="/var/lib/kubelet/pods/a8d299b1-ab44-421a-b922-bacef89a9084/volumes" Oct 30 21:45:19.993636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.993612 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" path="/var/lib/kubelet/pods/cafeeafb-8b05-4378-bb7f-102784b2d91b/volumes" Oct 30 21:45:19.999088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:19.999051 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv"] Oct 30 21:45:20.340522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.335957 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" event={"ID":"a7d44537-c273-47c3-b64e-b3225d200495","Type":"ContainerDied","Data":"1ae265306056a2c1db2723d050531628d1ec5f4c18241146c59e2cc60acee724"} Oct 30 21:45:20.340522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.336069 4526 scope.go:122] "RemoveContainer" containerID="71a30a9ba205787f35cd2d7726798931c82c0a57ebf75f8abd378cfc89970901" Oct 30 21:45:20.340522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.336222 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq" Oct 30 21:45:20.368944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.368911 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" event={"ID":"35eb2ffd-c29e-494a-8aa9-db939e290075","Type":"ContainerDied","Data":"9b185c79251d26faf1b703de219513627ec942827375fda1b993d8c0b2a7c0a3"} Oct 30 21:45:20.369165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.368981 4526 scope.go:122] "RemoveContainer" containerID="b174c1e7db62aa79fdf3353ca06a564f4a277f105264589797738661688377de" Oct 30 21:45:20.369165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.369092 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs" Oct 30 21:45:20.415519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.414586 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv"] Oct 30 21:45:20.518926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.518820 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr"] Oct 30 21:45:20.561269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.561233 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs"] Oct 30 21:45:20.588522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.583936 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-22qvs"] Oct 30 21:45:20.660342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.660312 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:20.713874 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.713823 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq"] Oct 30 21:45:20.718971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.718938 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9796f1-29889945-lv7kq"] Oct 30 21:45:20.787225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.787197 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:20.787665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.787645 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdef6a3-2a30-4bbe-ab10-64abc1239a66-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdef6a3-2a30-4bbe-ab10-64abc1239a66-maas-api-service-ca\") pod \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\" (UID: \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\") " Oct 30 21:45:20.787799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.787752 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdef6a3-2a30-4bbe-ab10-64abc1239a66-kube-api-access-dd6vk\" (UniqueName: \"kubernetes.io/projected/acdef6a3-2a30-4bbe-ab10-64abc1239a66-kube-api-access-dd6vk\") pod \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\" (UID: \"acdef6a3-2a30-4bbe-ab10-64abc1239a66\") " Oct 30 21:45:20.788006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.787986 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdef6a3-2a30-4bbe-ab10-64abc1239a66-maas-api-service-ca" pod "acdef6a3-2a30-4bbe-ab10-64abc1239a66" (UID: "acdef6a3-2a30-4bbe-ab10-64abc1239a66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.788365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.788347 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdef6a3-2a30-4bbe-ab10-64abc1239a66-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.790136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.790113 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdef6a3-2a30-4bbe-ab10-64abc1239a66-kube-api-access-dd6vk" pod "acdef6a3-2a30-4bbe-ab10-64abc1239a66" (UID: "acdef6a3-2a30-4bbe-ab10-64abc1239a66"). InnerVolumeSpecName "kube-api-access-dd6vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.889435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.889401 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd16d13-ff85-4eb2-a567-2800b627aa54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd16d13-ff85-4eb2-a567-2800b627aa54-maas-api-service-ca\") pod \"0cd16d13-ff85-4eb2-a567-2800b627aa54\" (UID: \"0cd16d13-ff85-4eb2-a567-2800b627aa54\") " Oct 30 21:45:20.889703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.889461 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd16d13-ff85-4eb2-a567-2800b627aa54-kube-api-access-kjrt2\" (UniqueName: \"kubernetes.io/projected/0cd16d13-ff85-4eb2-a567-2800b627aa54-kube-api-access-kjrt2\") pod \"0cd16d13-ff85-4eb2-a567-2800b627aa54\" (UID: \"0cd16d13-ff85-4eb2-a567-2800b627aa54\") " Oct 30 21:45:20.889920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.889889 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd16d13-ff85-4eb2-a567-2800b627aa54-maas-api-service-ca" pod "0cd16d13-ff85-4eb2-a567-2800b627aa54" (UID: "0cd16d13-ff85-4eb2-a567-2800b627aa54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.890295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.890277 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd16d13-ff85-4eb2-a567-2800b627aa54-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.890336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.890299 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6vk\" (UniqueName: \"kubernetes.io/projected/acdef6a3-2a30-4bbe-ab10-64abc1239a66-kube-api-access-dd6vk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:20.891820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.891798 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd16d13-ff85-4eb2-a567-2800b627aa54-kube-api-access-kjrt2" pod "0cd16d13-ff85-4eb2-a567-2800b627aa54" (UID: "0cd16d13-ff85-4eb2-a567-2800b627aa54"). InnerVolumeSpecName "kube-api-access-kjrt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.995577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:20.995529 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjrt2\" (UniqueName: \"kubernetes.io/projected/0cd16d13-ff85-4eb2-a567-2800b627aa54-kube-api-access-kjrt2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:21.401081 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.401051 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" Oct 30 21:45:21.401081 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.401061 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr" event={"ID":"0cd16d13-ff85-4eb2-a567-2800b627aa54","Type":"ContainerDied","Data":"e61f1d11cc0e66e99b63c88dde14f3bac6c1215e5771e07b6ec1df90534b547c"} Oct 30 21:45:21.401602 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.401122 4526 scope.go:122] "RemoveContainer" containerID="657e38d9f7c64016368ae71d564345b511469d4cb5772b48d0a31439abbd98f8" Oct 30 21:45:21.404542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.404460 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" event={"ID":"acdef6a3-2a30-4bbe-ab10-64abc1239a66","Type":"ContainerDied","Data":"97c81409b9cf92ea214611ab98716b3dd3bc33f6bdf91e1042fc74ddab31d316"} Oct 30 21:45:21.404784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.404708 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv" Oct 30 21:45:21.428775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.428746 4526 scope.go:122] "RemoveContainer" containerID="4d871a5c42bf42dcf7d5127bc4d1381f74599fd8a4b7a50f79107bd1063f291c" Oct 30 21:45:21.572542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.572462 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr"] Oct 30 21:45:21.577186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.577155 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-89c0fd-29889945-q7qwr"] Oct 30 21:45:21.681449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.681369 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv"] Oct 30 21:45:21.687270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.687237 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-021907-29889945-br4hv"] Oct 30 21:45:21.990614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.990524 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" path="/var/lib/kubelet/pods/0cd16d13-ff85-4eb2-a567-2800b627aa54/volumes" Oct 30 21:45:21.991288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.991265 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" path="/var/lib/kubelet/pods/35eb2ffd-c29e-494a-8aa9-db939e290075/volumes" Oct 30 21:45:21.991895 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.991875 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d44537-c273-47c3-b64e-b3225d200495" path="/var/lib/kubelet/pods/a7d44537-c273-47c3-b64e-b3225d200495/volumes" Oct 30 21:45:21.992860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:21.992842 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" path="/var/lib/kubelet/pods/acdef6a3-2a30-4bbe-ab10-64abc1239a66/volumes" Oct 30 21:45:22.050111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:22.050071 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n"] Oct 30 21:45:22.054444 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:22.054416 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ba9f570-vxw9n"] Oct 30 21:45:23.987796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:23.987761 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7418a23c-fab4-4512-89d2-fad697a23192" path="/var/lib/kubelet/pods/7418a23c-fab4-4512-89d2-fad697a23192/volumes" Oct 30 21:45:24.877239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:24.877202 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" podUID="3c437433-8510-422a-8ac3-ffba19fe30c8" containerName="payload-processing" containerID="cri-o://224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb" gracePeriod=25 Oct 30 21:45:24.999697 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:24.999653 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" podUID="e2793212-6363-4bff-be30-2ce58a0701bb" containerName="payload-pre-processing" containerID="cri-o://5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21" gracePeriod=25 Oct 30 21:45:25.200003 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.199612 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:45:25.293741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.279602 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c437433-8510-422a-8ac3-ffba19fe30c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c437433-8510-422a-8ac3-ffba19fe30c8-plugins-config-volume\") pod \"3c437433-8510-422a-8ac3-ffba19fe30c8\" (UID: \"3c437433-8510-422a-8ac3-ffba19fe30c8\") " Oct 30 21:45:25.293741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.279796 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c437433-8510-422a-8ac3-ffba19fe30c8-kube-api-access-jq9wl\" (UniqueName: \"kubernetes.io/projected/3c437433-8510-422a-8ac3-ffba19fe30c8-kube-api-access-jq9wl\") pod \"3c437433-8510-422a-8ac3-ffba19fe30c8\" (UID: \"3c437433-8510-422a-8ac3-ffba19fe30c8\") " Oct 30 21:45:25.293741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.281091 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c437433-8510-422a-8ac3-ffba19fe30c8-plugins-config-volume" pod "3c437433-8510-422a-8ac3-ffba19fe30c8" (UID: "3c437433-8510-422a-8ac3-ffba19fe30c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:25.293741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.291047 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c437433-8510-422a-8ac3-ffba19fe30c8-kube-api-access-jq9wl" pod "3c437433-8510-422a-8ac3-ffba19fe30c8" (UID: "3c437433-8510-422a-8ac3-ffba19fe30c8"). InnerVolumeSpecName "kube-api-access-jq9wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:25.317537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.317473 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:45:25.381815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.381784 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2793212-6363-4bff-be30-2ce58a0701bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2793212-6363-4bff-be30-2ce58a0701bb-plugins-config-volume\") pod \"e2793212-6363-4bff-be30-2ce58a0701bb\" (UID: \"e2793212-6363-4bff-be30-2ce58a0701bb\") " Oct 30 21:45:25.382001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.381839 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2793212-6363-4bff-be30-2ce58a0701bb-kube-api-access-rbm94\" (UniqueName: \"kubernetes.io/projected/e2793212-6363-4bff-be30-2ce58a0701bb-kube-api-access-rbm94\") pod \"e2793212-6363-4bff-be30-2ce58a0701bb\" (UID: \"e2793212-6363-4bff-be30-2ce58a0701bb\") " Oct 30 21:45:25.382356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.382325 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2793212-6363-4bff-be30-2ce58a0701bb-plugins-config-volume" pod "e2793212-6363-4bff-be30-2ce58a0701bb" (UID: "e2793212-6363-4bff-be30-2ce58a0701bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:25.382716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.382689 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c437433-8510-422a-8ac3-ffba19fe30c8-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.382849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.382718 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq9wl\" (UniqueName: \"kubernetes.io/projected/3c437433-8510-422a-8ac3-ffba19fe30c8-kube-api-access-jq9wl\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.382849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.382734 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2793212-6363-4bff-be30-2ce58a0701bb-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.384565 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.384543 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2793212-6363-4bff-be30-2ce58a0701bb-kube-api-access-rbm94" pod "e2793212-6363-4bff-be30-2ce58a0701bb" (UID: "e2793212-6363-4bff-be30-2ce58a0701bb"). InnerVolumeSpecName "kube-api-access-rbm94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:25.471904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.471807 4526 generic.go:356] "Generic (PLEG): container finished" podID="3c437433-8510-422a-8ac3-ffba19fe30c8" containerID="224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb" exitCode=0 Oct 30 21:45:25.471904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.471873 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" Oct 30 21:45:25.471904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.471878 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" event={"ID":"3c437433-8510-422a-8ac3-ffba19fe30c8","Type":"ContainerDied","Data":"224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb"} Oct 30 21:45:25.472169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.471961 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz" event={"ID":"3c437433-8510-422a-8ac3-ffba19fe30c8","Type":"ContainerDied","Data":"3297986faef598f23d35c7fee4f4972ef1ec7e1ad5a9231b60b9e030ef35663e"} Oct 30 21:45:25.472169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.471984 4526 scope.go:122] "RemoveContainer" containerID="224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb" Oct 30 21:45:25.473654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.473632 4526 generic.go:356] "Generic (PLEG): container finished" podID="e2793212-6363-4bff-be30-2ce58a0701bb" containerID="5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21" exitCode=0 Oct 30 21:45:25.473801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.473735 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" Oct 30 21:45:25.473870 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.473806 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" event={"ID":"e2793212-6363-4bff-be30-2ce58a0701bb","Type":"ContainerDied","Data":"5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21"} Oct 30 21:45:25.473870 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.473853 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv" event={"ID":"e2793212-6363-4bff-be30-2ce58a0701bb","Type":"ContainerDied","Data":"198015cfc1f0adac31e435ad3bd091fd216750b423acdf4b4df0de72a85f39c4"} Oct 30 21:45:25.484424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.484393 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbm94\" (UniqueName: \"kubernetes.io/projected/e2793212-6363-4bff-be30-2ce58a0701bb-kube-api-access-rbm94\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.486351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.486238 4526 scope.go:122] "RemoveContainer" containerID="224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb" Oct 30 21:45:25.486655 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:25.486630 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb\": container with ID starting with 224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb not found: ID does not exist" containerID="224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb" Oct 30 21:45:25.486745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.486669 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb"} err="failed to get container status \"224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb\": rpc error: code = NotFound desc = could not find container \"224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb\": container with ID starting with 224622dd761c4ca59e3ccae99097ff8cc3cded22986b349bf8ab2aa870ee58fb not found: ID does not exist" Oct 30 21:45:25.486745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.486697 4526 scope.go:122] "RemoveContainer" containerID="5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21" Oct 30 21:45:25.500810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.500786 4526 scope.go:122] "RemoveContainer" containerID="5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21" Oct 30 21:45:25.501172 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:25.501148 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21\": container with ID starting with 5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21 not found: ID does not exist" containerID="5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21" Oct 30 21:45:25.501249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.501186 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21"} err="failed to get container status \"5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21\": rpc error: code = NotFound desc = could not find container \"5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21\": container with ID starting with 5be7c11191ce067cc3efcb324bcd047c410855ca7e14a9499653f4d877553f21 not found: ID does not exist" Oct 30 21:45:25.645065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.644985 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv"] Oct 30 21:45:25.650412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.650383 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1915737c-c7894687f-wtnxv"] Oct 30 21:45:25.762949 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.762818 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz"] Oct 30 21:45:25.772786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.772757 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1915737c-9d5ffb579-5fzrz"] Oct 30 21:45:25.987305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.987271 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c437433-8510-422a-8ac3-ffba19fe30c8" path="/var/lib/kubelet/pods/3c437433-8510-422a-8ac3-ffba19fe30c8/volumes" Oct 30 21:45:25.987834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:25.987818 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2793212-6363-4bff-be30-2ce58a0701bb" path="/var/lib/kubelet/pods/e2793212-6363-4bff-be30-2ce58a0701bb/volumes" Oct 30 21:45:29.056718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:29.056677 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m"] Oct 30 21:45:29.136532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:29.135924 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae7eab26-vff6m"] Oct 30 21:45:29.987830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:29.987791 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d141cab6-5b5b-4a79-bfc4-126ae900563e" path="/var/lib/kubelet/pods/d141cab6-5b5b-4a79-bfc4-126ae900563e/volumes" Oct 30 21:45:36.179712 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:36.179660 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:45:36.180248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:36.179759 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:45:38.142935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.089436 4526 patch_prober.go:29] interesting pod/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": read tcp 10.133.0.2:52000->10.133.0.154:15021: read: connection reset by peer" start-of-body= Oct 30 21:45:38.142935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.089519 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": read tcp 10.133.0.2:52000->10.133.0.154:15021: read: connection reset by peer" Oct 30 21:45:38.575720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.575687 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf"] Oct 30 21:45:38.576795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.575883 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" containerID="cri-o://6161c51f505c1483c44be155e91d46786b2ddf0263917ce8a37afbd66dfba50e" gracePeriod=30 Oct 30 21:45:38.625815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.625779 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp"] Oct 30 21:45:38.626078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.626035 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" containerID="cri-o://3c040f292e80f56c2fd164d1c5c2368a5764541eff2d8d04bd7996c166348405" gracePeriod=30 Oct 30 21:45:38.632052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.632028 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8"] Oct 30 21:45:38.632284 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.632261 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" containerID="cri-o://1257e621b3e782f32dcbde87e10c305b07af8fb00416fd674d64f0e5e81e6c6a" gracePeriod=30 Oct 30 21:45:38.671610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.671575 4526 generic.go:356] "Generic (PLEG): container finished" podID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerID="fdb610614e5912c4863a59bc3572df620dda014d9aab572e68590ed7bca497c1" exitCode=0 Oct 30 21:45:38.671813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.671615 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" event={"ID":"f4dc33b1-e449-4b7f-b1eb-10adff633529","Type":"ContainerDied","Data":"fdb610614e5912c4863a59bc3572df620dda014d9aab572e68590ed7bca497c1"} Oct 30 21:45:38.672161 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.672122 4526 scope.go:122] "RemoveContainer" containerID="fdb610614e5912c4863a59bc3572df620dda014d9aab572e68590ed7bca497c1" Oct 30 21:45:38.678585 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.674480 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:38.678742 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.678682 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:38.678820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.678757 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:38.679794 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.679770 4526 generic.go:356] "Generic (PLEG): container finished" podID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerID="9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa" exitCode=0 Oct 30 21:45:38.680244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.680207 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" event={"ID":"0cc8fdad-213d-4748-94f3-f4c1359c8bbc","Type":"ContainerDied","Data":"9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa"} Oct 30 21:45:38.680896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.680866 4526 scope.go:122] "RemoveContainer" containerID="9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa" Oct 30 21:45:38.682870 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.682842 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:38.682962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.682924 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:38.683027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:38.682971 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:39.735761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:39.735715 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" event={"ID":"0cc8fdad-213d-4748-94f3-f4c1359c8bbc","Type":"ContainerStarted","Data":"61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279"} Oct 30 21:45:39.739530 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:39.739476 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" event={"ID":"f4dc33b1-e449-4b7f-b1eb-10adff633529","Type":"ContainerStarted","Data":"86bcd892373262f756df14a998a1bb7d137b616332366ae2158de3d20f60208f"} Oct 30 21:45:39.868499 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:39.868457 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:45:39.868718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:39.868631 4526 patch_prober.go:29] interesting pod/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:45:39.868718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:39.868686 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 21:45:40.552180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:40.552145 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:45:40.552568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:40.552546 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:45:40.552621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:40.552600 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 21:45:40.869247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:40.869155 4526 patch_prober.go:29] interesting pod/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:45:40.869247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:40.869205 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 21:45:41.552581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:41.552540 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:45:41.552806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:41.552600 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 21:45:41.868601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:41.868506 4526 patch_prober.go:29] interesting pod/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:45:41.868601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:41.868565 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 21:45:42.480296 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:45:42.480259 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:45:42.551998 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:42.551962 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:45:42.552182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:42.552038 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 21:45:42.868519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:42.868471 4526 patch_prober.go:29] interesting pod/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:45:42.868729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:42.868546 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 21:45:43.552143 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.552110 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:45:43.552585 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.552164 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 21:45:43.752777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.752743 4526 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:45:43.752957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.752816 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 21:45:43.792510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.791312 4526 generic.go:356] "Generic (PLEG): container finished" podID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerID="1257e621b3e782f32dcbde87e10c305b07af8fb00416fd674d64f0e5e81e6c6a" exitCode=0 Oct 30 21:45:43.792510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.791473 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" event={"ID":"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da","Type":"ContainerDied","Data":"1257e621b3e782f32dcbde87e10c305b07af8fb00416fd674d64f0e5e81e6c6a"} Oct 30 21:45:43.800513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.793844 4526 generic.go:356] "Generic (PLEG): container finished" podID="a502da11-e60f-46c3-9d00-22c05e102c21" containerID="6161c51f505c1483c44be155e91d46786b2ddf0263917ce8a37afbd66dfba50e" exitCode=0 Oct 30 21:45:43.800513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.794045 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" event={"ID":"a502da11-e60f-46c3-9d00-22c05e102c21","Type":"ContainerDied","Data":"6161c51f505c1483c44be155e91d46786b2ddf0263917ce8a37afbd66dfba50e"} Oct 30 21:45:43.838568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.837480 4526 generic.go:356] "Generic (PLEG): container finished" podID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerID="3c040f292e80f56c2fd164d1c5c2368a5764541eff2d8d04bd7996c166348405" exitCode=0 Oct 30 21:45:43.838568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.837596 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" event={"ID":"30832c28-5ab0-45af-afd4-54360c2e0c2e","Type":"ContainerDied","Data":"3c040f292e80f56c2fd164d1c5c2368a5764541eff2d8d04bd7996c166348405"} Oct 30 21:45:43.877909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.877251 4526 patch_prober.go:29] interesting pod/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:45:43.877909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:43.877307 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 21:45:44.167252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.167219 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:45:44.264780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.264744 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-socket\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.264987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.264848 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istiod-ca-cert\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.264987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.264878 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-certs\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.265107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.264987 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-data\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.265526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.265473 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istiod-ca-cert" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:44.266154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.265917 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-certs" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.266248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.266122 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-data" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.266248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.266214 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-socket" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.266358 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.266293 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-envoy\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.266503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.266466 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-kube-api-access-scchl\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-kube-api-access-scchl\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.266588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.266538 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-credential-socket\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.266681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.266592 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-podinfo\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.266751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.266686 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-token\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-token\") pod \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\" (UID: \"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da\") " Oct 30 21:45:44.268074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.268051 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-credential-socket" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.269006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.268987 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.269121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.269108 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.269216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.269204 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.269298 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.269287 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.269411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.269399 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.280506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.278598 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-podinfo" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:45:44.280506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.278788 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-envoy" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.282614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.282532 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-token" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:44.283247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.283208 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-kube-api-access-scchl" pod "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" (UID: "b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da"). InnerVolumeSpecName "kube-api-access-scchl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:44.363761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.363735 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:45:44.368035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.368013 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:45:44.371842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.371819 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scchl\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-kube-api-access-scchl\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.371964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.371843 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.371964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.371855 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.371964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.371864 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.472728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.472633 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-data\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.472917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.472742 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-kube-api-access-95lbd\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-kube-api-access-95lbd\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.472917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.472781 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-podinfo\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.472917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.472810 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a502da11-e60f-46c3-9d00-22c05e102c21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a502da11-e60f-46c3-9d00-22c05e102c21-istiod-ca-cert\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.472917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.472841 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-kube-api-access-hmthw\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-kube-api-access-hmthw\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.473153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.472921 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-certs\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.473153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.472937 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-data" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.473153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.472976 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-data\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.473153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473008 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-socket\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.473153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473055 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-istio-token\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-istio-token\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.473409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473184 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-envoy\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.473409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473252 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30832c28-5ab0-45af-afd4-54360c2e0c2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30832c28-5ab0-45af-afd4-54360c2e0c2e-istiod-ca-cert\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.473409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473274 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-credential-socket\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.473409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473311 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-certs\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.473409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473363 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a502da11-e60f-46c3-9d00-22c05e102c21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a502da11-e60f-46c3-9d00-22c05e102c21-istio-podinfo\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.473409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473384 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-envoy\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.473740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473471 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-credential-socket\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.473740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473553 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-socket\") pod \"a502da11-e60f-46c3-9d00-22c05e102c21\" (UID: \"a502da11-e60f-46c3-9d00-22c05e102c21\") " Oct 30 21:45:44.473740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.473584 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-token\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-token\") pod \"30832c28-5ab0-45af-afd4-54360c2e0c2e\" (UID: \"30832c28-5ab0-45af-afd4-54360c2e0c2e\") " Oct 30 21:45:44.474209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.474192 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.475305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.475098 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-certs" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.475608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.475466 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a502da11-e60f-46c3-9d00-22c05e102c21-istiod-ca-cert" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:44.475608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.475541 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-socket" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.475933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.475819 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-podinfo" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:45:44.476043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.476000 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-data" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.476336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.476199 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-credential-socket" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.476336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.476264 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-credential-socket" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.476783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.476760 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30832c28-5ab0-45af-afd4-54360c2e0c2e-istiod-ca-cert" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:44.477883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.477552 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-kube-api-access-95lbd" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "kube-api-access-95lbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:44.477883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.477825 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-envoy" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.477883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.477850 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-socket" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.478214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.478186 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-certs" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.478430 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.478405 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-istio-token" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:44.478542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.478523 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-kube-api-access-hmthw" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "kube-api-access-hmthw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:44.478749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.478727 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a502da11-e60f-46c3-9d00-22c05e102c21-istio-podinfo" pod "a502da11-e60f-46c3-9d00-22c05e102c21" (UID: "a502da11-e60f-46c3-9d00-22c05e102c21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:45:44.478905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.478882 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-envoy" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:45:44.479324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.479309 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-token" pod "30832c28-5ab0-45af-afd4-54360c2e0c2e" (UID: "30832c28-5ab0-45af-afd4-54360c2e0c2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:44.560128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.560088 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:45:44.560669 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.560153 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 21:45:44.575858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.575822 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576085 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a502da11-e60f-46c3-9d00-22c05e102c21-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576210 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576295 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576380 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576460 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576568 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lbd\" (UniqueName: \"kubernetes.io/projected/30832c28-5ab0-45af-afd4-54360c2e0c2e-kube-api-access-95lbd\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576662 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30832c28-5ab0-45af-afd4-54360c2e0c2e-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576752 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a502da11-e60f-46c3-9d00-22c05e102c21-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576834 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmthw\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-kube-api-access-hmthw\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.576930 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.576920 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.577018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.577008 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.577099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.577088 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30832c28-5ab0-45af-afd4-54360c2e0c2e-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.577183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.577172 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a502da11-e60f-46c3-9d00-22c05e102c21-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.577262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.577252 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.577341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.577331 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30832c28-5ab0-45af-afd4-54360c2e0c2e-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.577419 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.577409 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a502da11-e60f-46c3-9d00-22c05e102c21-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:44.873527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.873380 4526 patch_prober.go:29] interesting pod/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:45:44.873902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.873536 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 21:45:44.952528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.946933 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" event={"ID":"30832c28-5ab0-45af-afd4-54360c2e0c2e","Type":"ContainerDied","Data":"81641d18365038cb364bdb6be99e228ef3f7ce9207b48b68b8408b0f0ae379e0"} Oct 30 21:45:44.952528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.947009 4526 scope.go:122] "RemoveContainer" containerID="3c040f292e80f56c2fd164d1c5c2368a5764541eff2d8d04bd7996c166348405" Oct 30 21:45:44.952528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.947192 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp" Oct 30 21:45:44.987513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.969419 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" event={"ID":"b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da","Type":"ContainerDied","Data":"5e2ce7da469b351e1ea67df61d37be9d32c1af9bd7c2cb1241b9997d74096738"} Oct 30 21:45:44.987513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.970106 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8" Oct 30 21:45:44.987513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.984594 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" event={"ID":"a502da11-e60f-46c3-9d00-22c05e102c21","Type":"ContainerDied","Data":"327ddcb29b16c22fafe1674ac8b6cd0015558908e19fea0bd274b79177f57a84"} Oct 30 21:45:44.987513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:44.984774 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf" Oct 30 21:45:45.064420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.064398 4526 scope.go:122] "RemoveContainer" containerID="1257e621b3e782f32dcbde87e10c305b07af8fb00416fd674d64f0e5e81e6c6a" Oct 30 21:45:45.076214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.076191 4526 scope.go:122] "RemoveContainer" containerID="6161c51f505c1483c44be155e91d46786b2ddf0263917ce8a37afbd66dfba50e" Oct 30 21:45:45.233424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.233337 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8"] Oct 30 21:45:45.238159 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.238128 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-792g8"] Oct 30 21:45:45.345433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.345404 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp"] Oct 30 21:45:45.349632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.349606 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-zb4hp"] Oct 30 21:45:45.456030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.455991 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf"] Oct 30 21:45:45.462251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.462220 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-rj6gf"] Oct 30 21:45:45.556853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.556822 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:45:45.557162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.557138 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:45:45.557889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.557870 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:45:45.873019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.872948 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:45:45.873419 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.873406 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:45:45.874168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.874151 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:45:45.988875 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.988845 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" path="/var/lib/kubelet/pods/30832c28-5ab0-45af-afd4-54360c2e0c2e/volumes" Oct 30 21:45:45.989416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.989402 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" path="/var/lib/kubelet/pods/a502da11-e60f-46c3-9d00-22c05e102c21/volumes" Oct 30 21:45:45.989887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:45.989873 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" path="/var/lib/kubelet/pods/b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da/volumes" Oct 30 21:45:51.014831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.014804 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf"] Oct 30 21:45:51.022928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.022908 4526 scope.go:122] "RemoveContainer" containerID="24b0c6e7caf98f0afb6c4fcac6d7740c458916ac9617502cea1a87e9980d3971" Oct 30 21:45:51.034526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.034499 4526 scope.go:122] "RemoveContainer" containerID="4ef51a2f239ba57464a23424515cb7c2a8b30d9fa5494cfd5731ba8a768050f3" Oct 30 21:45:51.045191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.045174 4526 scope.go:122] "RemoveContainer" containerID="623702df91ad30033987994d6dabcfeab8c6ed93cfa557cae575ddd7b1f9e361" Oct 30 21:45:51.055157 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.055138 4526 scope.go:122] "RemoveContainer" containerID="dcc1f23f89e371102fa418c12e893d25035b922c7e3ff1432674c46f35a6ddca" Oct 30 21:45:51.065147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.065123 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9173600a-s88zf"] Oct 30 21:45:51.066579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.066569 4526 scope.go:122] "RemoveContainer" containerID="592d9707ee1149df8e8abbd2a3acbb90d5205765f1cd1310153fa0c0f40780e5" Oct 30 21:45:51.077479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.077459 4526 scope.go:122] "RemoveContainer" containerID="a4fe20f8f878e05dfd06c8d1173e4be10eb122ca497370ec0d5345c7234daf30" Oct 30 21:45:51.088731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.088710 4526 scope.go:122] "RemoveContainer" containerID="28754d3c4cd76f8ae22faa856455b523a39170c475a4fb9a4ee8d544c1bb82d8" Oct 30 21:45:51.098281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.098261 4526 scope.go:122] "RemoveContainer" containerID="f8429e299789bb324dc918ac975b90d8bc9923903c01a1877925be4588fd30f2" Oct 30 21:45:51.109356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.109338 4526 scope.go:122] "RemoveContainer" containerID="f53704ebe574a0f89c1a83dfd43515374a8186a978ea01011822eee9f829df1a" Oct 30 21:45:51.119047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.119026 4526 scope.go:122] "RemoveContainer" containerID="9fbc19b1cac1e0adcc67ba73917291ba48506f6fd3d40a3fda949fdc8829ef68" Oct 30 21:45:51.128239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.128220 4526 scope.go:122] "RemoveContainer" containerID="1f65ac5708470d9890d9c94caa0959b18199835b3e8caa21204dc6bbfcc00ca1" Oct 30 21:45:51.137089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.137072 4526 scope.go:122] "RemoveContainer" containerID="9baa1a0a451002f866f90d03e3b5e23d82a0a2d6ccbe20ff3430e44fe59d6155" Oct 30 21:45:51.145792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.145773 4526 scope.go:122] "RemoveContainer" containerID="e839a2f4f9dff60b2994e89f909af227ea569308d75ab44e4d69b7995180d6fa" Oct 30 21:45:51.154482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.154466 4526 scope.go:122] "RemoveContainer" containerID="bdeef5e74091189a6f47f62456166dbc145e2d912d4b772adfc7f757afdef7c6" Oct 30 21:45:51.163286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.163269 4526 scope.go:122] "RemoveContainer" containerID="5d5c80ef69d69e8b295c7ea4e8fc2f42266c02c6d99251459df5aacbde67fda3" Oct 30 21:45:51.987227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:51.987183 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0dc8bad-3f60-4fbe-bac4-006595417e45" path="/var/lib/kubelet/pods/e0dc8bad-3f60-4fbe-bac4-006595417e45/volumes" Oct 30 21:45:54.853647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.853615 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd"] Oct 30 21:45:54.854576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854554 4526 cpu_manager.go:414] "containerMap: removing container" podUID="6aa12cd0-2d28-4dc9-988f-687b912375e5" containerName="revoke-keys" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854576 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa12cd0-2d28-4dc9-988f-687b912375e5" containerName="revoke-keys" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854592 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854599 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854614 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854620 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854644 4526 cpu_manager.go:414] "containerMap: removing container" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854651 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854659 4526 cpu_manager.go:414] "containerMap: removing container" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854667 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854679 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854687 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854698 4526 cpu_manager.go:414] "containerMap: removing container" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:45:54.854701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854705 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854715 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854722 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854738 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3c437433-8510-422a-8ac3-ffba19fe30c8" containerName="payload-processing" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854746 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c437433-8510-422a-8ac3-ffba19fe30c8" containerName="payload-processing" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854759 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854766 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854783 4526 cpu_manager.go:414] "containerMap: removing container" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854790 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854804 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854810 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854829 4526 cpu_manager.go:414] "containerMap: removing container" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854836 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854855 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854861 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854873 4526 cpu_manager.go:414] "containerMap: removing container" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854880 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854888 4526 cpu_manager.go:414] "containerMap: removing container" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854895 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854906 4526 cpu_manager.go:414] "containerMap: removing container" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854914 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854929 4526 cpu_manager.go:414] "containerMap: removing container" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854936 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854953 4526 cpu_manager.go:414] "containerMap: removing container" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854960 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854974 4526 cpu_manager.go:414] "containerMap: removing container" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854981 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854992 4526 cpu_manager.go:414] "containerMap: removing container" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.854999 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855012 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e2793212-6363-4bff-be30-2ce58a0701bb" containerName="payload-pre-processing" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855019 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2793212-6363-4bff-be30-2ce58a0701bb" containerName="payload-pre-processing" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855030 4526 cpu_manager.go:414] "containerMap: removing container" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855037 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855211 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a502da11-e60f-46c3-9d00-22c05e102c21" containerName="istio-proxy" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855228 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855237 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="b262a6c9-4ea9-4b7f-9e4d-c1fa65d196da" containerName="istio-proxy" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855247 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855257 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855265 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855274 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855283 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c437433-8510-422a-8ac3-ffba19fe30c8" containerName="payload-processing" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855291 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855300 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855308 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855317 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855329 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855337 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855347 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855355 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855364 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:45:54.855352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855373 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:45:54.857726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855384 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.857726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855394 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:45:54.857726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855422 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="30832c28-5ab0-45af-afd4-54360c2e0c2e" containerName="istio-proxy" Oct 30 21:45:54.857726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855434 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.857726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855443 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2793212-6363-4bff-be30-2ce58a0701bb" containerName="payload-pre-processing" Oct 30 21:45:54.857726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855452 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="17547078-dc7b-473d-ac5e-1cd327ffac61" containerName="cleanup" Oct 30 21:45:54.857726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.855462 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa12cd0-2d28-4dc9-988f-687b912375e5" containerName="revoke-keys" Oct 30 21:45:54.875272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.875246 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:54.939520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:54.935925 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-218800-openshift-default-dockercfg-696vk\"" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.022668 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-data\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.022745 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-token\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.022824 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istiod-ca-cert\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.022857 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-certs\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.022935 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-credential-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.022964 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.022992 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmcr\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-kube-api-access-kqmcr\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.023121 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-podinfo\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.023218 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-envoy\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.098645 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd"] Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.130773 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-podinfo\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.130902 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-envoy\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.130970 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-data\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.131032 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-token\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.131196 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istiod-ca-cert\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.131234 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-certs\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.131360 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-credential-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.131392 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.131418 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmcr\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-kube-api-access-kqmcr\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.132630 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-certs\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.133183 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-credential-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.133668 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.134191 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-data\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.134812 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istiod-ca-cert\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.138814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.135718 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-envoy\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.145293 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.145084 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-podinfo\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.189304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.189271 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmcr\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-kube-api-access-kqmcr\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.190786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.190761 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-token\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.196763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.196742 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:55.643195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.643160 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd"] Oct 30 21:45:55.680641 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:45:55.680604 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaba0bf1_59e8_4cd1_ab2f_173d71fdf7fb.slice/crio-10e9712db5434df6008321f5e751fc4a516928bf5264707a1dd28553fdc922d9 WatchSource:0}: Error finding container 10e9712db5434df6008321f5e751fc4a516928bf5264707a1dd28553fdc922d9: Status 404 returned error can't find the container with id 10e9712db5434df6008321f5e751fc4a516928bf5264707a1dd28553fdc922d9 Oct 30 21:45:55.683389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.683352 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:55.683507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.683428 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:55.683507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:55.683460 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:56.105220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:56.105187 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" event={"ID":"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb","Type":"ContainerStarted","Data":"0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58"} Oct 30 21:45:56.105776 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:56.105238 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" event={"ID":"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb","Type":"ContainerStarted","Data":"10e9712db5434df6008321f5e751fc4a516928bf5264707a1dd28553fdc922d9"} Oct 30 21:45:56.197101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:56.197065 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:56.203400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:56.203377 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:56.226993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:56.226937 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" podStartSLOduration=2.226923498 podStartE2EDuration="2.226923498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:56.22338921 +0000 UTC m=+1746.356762409" watchObservedRunningTime="2026-10-30 21:45:56.226923498 +0000 UTC m=+1746.360296696" Oct 30 21:45:56.893589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:56.893557 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9"] Oct 30 21:45:56.893795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:56.893760 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" containerID="cri-o://5faca56c00f6f1063238fea6620a22ca2ba12d9c0705a9deeed4573e7e3c4643" gracePeriod=30 Oct 30 21:45:57.113045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:57.113017 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:45:57.114620 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:45:57.114598 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:46:02.175669 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.175638 4526 generic.go:356] "Generic (PLEG): container finished" podID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerID="5faca56c00f6f1063238fea6620a22ca2ba12d9c0705a9deeed4573e7e3c4643" exitCode=0 Oct 30 21:46:02.176231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.175708 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" event={"ID":"75f0b7ee-7835-4c92-829e-f734300c96b5","Type":"ContainerDied","Data":"5faca56c00f6f1063238fea6620a22ca2ba12d9c0705a9deeed4573e7e3c4643"} Oct 30 21:46:02.176231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.175756 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" event={"ID":"75f0b7ee-7835-4c92-829e-f734300c96b5","Type":"ContainerDied","Data":"58ff6df9dd7e0e4696b410a45f123cf78ec4275a89cc07cdb542fc2f652edc00"} Oct 30 21:46:02.176231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.175777 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58ff6df9dd7e0e4696b410a45f123cf78ec4275a89cc07cdb542fc2f652edc00" Oct 30 21:46:02.193444 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.193418 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:46:02.275349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275305 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-socket\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.275563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275369 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-kube-api-access-98tk9\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-kube-api-access-98tk9\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.275563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275418 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-envoy\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.275700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275556 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-certs\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.275755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275716 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f0b7ee-7835-4c92-829e-f734300c96b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0b7ee-7835-4c92-829e-f734300c96b5-istiod-ca-cert\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.275809 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275762 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-token\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-token\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.275867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275820 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-certs" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.275940 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275889 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-socket" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.275993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.275944 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-podinfo\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.276049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.276030 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f0b7ee-7835-4c92-829e-f734300c96b5-istiod-ca-cert" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:02.276113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.276057 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-credential-socket\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.276167 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.276110 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-data\") pod \"75f0b7ee-7835-4c92-829e-f734300c96b5\" (UID: \"75f0b7ee-7835-4c92-829e-f734300c96b5\") " Oct 30 21:46:02.276386 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.276365 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-credential-socket" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.276856 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.276718 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-data" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.277093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.277071 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.277176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.277100 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.277176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.277115 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f0b7ee-7835-4c92-829e-f734300c96b5-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.277176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.277131 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.277176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.277145 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.278118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.278092 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-kube-api-access-98tk9" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "kube-api-access-98tk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.278388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.278365 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-envoy" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.278637 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.278612 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-token" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.279011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.278991 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-podinfo" pod "75f0b7ee-7835-4c92-829e-f734300c96b5" (UID: "75f0b7ee-7835-4c92-829e-f734300c96b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:02.379400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.379363 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.379400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.379391 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98tk9\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-kube-api-access-98tk9\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.379400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.379402 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.379400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.379411 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f0b7ee-7835-4c92-829e-f734300c96b5-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.842711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.842669 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg"] Oct 30 21:46:02.843557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843507 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:46:02.843557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843530 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843564 4526 cpu_manager.go:414] "containerMap: removing container" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843571 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843584 4526 cpu_manager.go:414] "containerMap: removing container" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843590 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafeeafb-8b05-4378-bb7f-102784b2d91b" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843599 4526 cpu_manager.go:414] "containerMap: removing container" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843606 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f63cc29-b531-4c4f-af3e-81a3704576ff" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843626 4526 cpu_manager.go:414] "containerMap: removing container" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843632 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843661 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843667 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843698 4526 cpu_manager.go:414] "containerMap: removing container" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843707 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843727 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:46:02.843743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843734 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:46:02.844401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843963 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d44537-c273-47c3-b64e-b3225d200495" containerName="cleanup" Oct 30 21:46:02.844401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843982 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="35eb2ffd-c29e-494a-8aa9-db939e290075" containerName="cleanup" Oct 30 21:46:02.844401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.843995 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" containerName="istio-proxy" Oct 30 21:46:02.844401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.844011 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d299b1-ab44-421a-b922-bacef89a9084" containerName="cleanup" Oct 30 21:46:02.844401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.844022 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd16d13-ff85-4eb2-a567-2800b627aa54" containerName="cleanup" Oct 30 21:46:02.844401 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.844034 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdef6a3-2a30-4bbe-ab10-64abc1239a66" containerName="cleanup" Oct 30 21:46:02.858775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.858741 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:02.952513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.932721 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-218800-dockercfg-fw4jg\"" Oct 30 21:46:02.952513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.932843 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-218800\"" Oct 30 21:46:02.952513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.933120 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg"] Oct 30 21:46:03.001586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.992908 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1afd7-9add-425e-b286-04a977e12935-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg\" (UID: \"98a1afd7-9add-425e-b286-04a977e12935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:03.001586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:02.993014 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8cmf\" (UniqueName: \"kubernetes.io/projected/98a1afd7-9add-425e-b286-04a977e12935-kube-api-access-g8cmf\") pod \"payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg\" (UID: \"98a1afd7-9add-425e-b286-04a977e12935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:03.097178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.082856 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9"] Oct 30 21:46:03.097178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.089109 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:03.126508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.121008 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9"] Oct 30 21:46:03.126508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.124222 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1afd7-9add-425e-b286-04a977e12935-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg\" (UID: \"98a1afd7-9add-425e-b286-04a977e12935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:03.126508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.124348 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8cmf\" (UniqueName: \"kubernetes.io/projected/98a1afd7-9add-425e-b286-04a977e12935-kube-api-access-g8cmf\") pod \"payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg\" (UID: \"98a1afd7-9add-425e-b286-04a977e12935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:03.129544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.128130 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1afd7-9add-425e-b286-04a977e12935-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg\" (UID: \"98a1afd7-9add-425e-b286-04a977e12935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:03.195516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.188082 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8cmf\" (UniqueName: \"kubernetes.io/projected/98a1afd7-9add-425e-b286-04a977e12935-kube-api-access-g8cmf\") pod \"payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg\" (UID: \"98a1afd7-9add-425e-b286-04a977e12935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:03.202617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.198752 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9" Oct 30 21:46:03.226547 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.226459 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstrg\" (UniqueName: \"kubernetes.io/projected/8637d261-f265-4b78-ae25-de43f0db20a6-kube-api-access-xstrg\") pod \"payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9\" (UID: \"8637d261-f265-4b78-ae25-de43f0db20a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:03.226547 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.226527 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8637d261-f265-4b78-ae25-de43f0db20a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9\" (UID: \"8637d261-f265-4b78-ae25-de43f0db20a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:03.328672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.328638 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstrg\" (UniqueName: \"kubernetes.io/projected/8637d261-f265-4b78-ae25-de43f0db20a6-kube-api-access-xstrg\") pod \"payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9\" (UID: \"8637d261-f265-4b78-ae25-de43f0db20a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:03.328900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.328687 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8637d261-f265-4b78-ae25-de43f0db20a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9\" (UID: \"8637d261-f265-4b78-ae25-de43f0db20a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:03.329572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.329546 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8637d261-f265-4b78-ae25-de43f0db20a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9\" (UID: \"8637d261-f265-4b78-ae25-de43f0db20a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:03.372047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.371968 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9"] Oct 30 21:46:03.376279 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.376249 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-7qvf9"] Oct 30 21:46:03.385741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.385716 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstrg\" (UniqueName: \"kubernetes.io/projected/8637d261-f265-4b78-ae25-de43f0db20a6-kube-api-access-xstrg\") pod \"payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9\" (UID: \"8637d261-f265-4b78-ae25-de43f0db20a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:03.445422 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.445381 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:03.474019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.473985 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:03.834141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.834114 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9"] Oct 30 21:46:03.834502 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:46:03.834459 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8637d261_f265_4b78_ae25_de43f0db20a6.slice/crio-fe2bb52dc12951e87b771a3aca5483daa92e97b75c5921118d70dd1c6e50ac6f WatchSource:0}: Error finding container fe2bb52dc12951e87b771a3aca5483daa92e97b75c5921118d70dd1c6e50ac6f: Status 404 returned error can't find the container with id fe2bb52dc12951e87b771a3aca5483daa92e97b75c5921118d70dd1c6e50ac6f Oct 30 21:46:03.883966 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:46:03.883940 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a1afd7_9add_425e_b286_04a977e12935.slice/crio-56fa49fb766937433f7e6d7b9ca0b25d9ea0087c7b4704f3f171055b54e75d6a WatchSource:0}: Error finding container 56fa49fb766937433f7e6d7b9ca0b25d9ea0087c7b4704f3f171055b54e75d6a: Status 404 returned error can't find the container with id 56fa49fb766937433f7e6d7b9ca0b25d9ea0087c7b4704f3f171055b54e75d6a Oct 30 21:46:03.884133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.884110 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg"] Oct 30 21:46:03.987471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:03.987439 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f0b7ee-7835-4c92-829e-f734300c96b5" path="/var/lib/kubelet/pods/75f0b7ee-7835-4c92-829e-f734300c96b5/volumes" Oct 30 21:46:04.210030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.209941 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" event={"ID":"98a1afd7-9add-425e-b286-04a977e12935","Type":"ContainerStarted","Data":"87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5"} Oct 30 21:46:04.210030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.210002 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" event={"ID":"98a1afd7-9add-425e-b286-04a977e12935","Type":"ContainerStarted","Data":"56fa49fb766937433f7e6d7b9ca0b25d9ea0087c7b4704f3f171055b54e75d6a"} Oct 30 21:46:04.210595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.210188 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:04.212009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.211986 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" event={"ID":"8637d261-f265-4b78-ae25-de43f0db20a6","Type":"ContainerStarted","Data":"ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710"} Oct 30 21:46:04.212127 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.212028 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" event={"ID":"8637d261-f265-4b78-ae25-de43f0db20a6","Type":"ContainerStarted","Data":"fe2bb52dc12951e87b771a3aca5483daa92e97b75c5921118d70dd1c6e50ac6f"} Oct 30 21:46:04.212127 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.212056 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:04.327778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.327724 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" podStartSLOduration=2.327710748 podStartE2EDuration="2.327710748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:04.324173509 +0000 UTC m=+1754.457546711" watchObservedRunningTime="2026-10-30 21:46:04.327710748 +0000 UTC m=+1754.461083946" Oct 30 21:46:04.454181 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.449266 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" podStartSLOduration=2.449240081 podStartE2EDuration="2.449240081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:04.442124286 +0000 UTC m=+1754.575497496" watchObservedRunningTime="2026-10-30 21:46:04.449240081 +0000 UTC m=+1754.582613280" Oct 30 21:46:04.597761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.597551 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2"] Oct 30 21:46:04.597973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.597932 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" containerID="cri-o://79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec" gracePeriod=30 Oct 30 21:46:04.607047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.603572 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl"] Oct 30 21:46:04.607047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.603820 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" containerID="cri-o://38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a" gracePeriod=30 Oct 30 21:46:04.608902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.608871 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7"] Oct 30 21:46:04.609531 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.609399 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" containerID="cri-o://69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee" gracePeriod=30 Oct 30 21:46:04.613458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.613435 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg"] Oct 30 21:46:04.613693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:04.613649 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" containerID="cri-o://1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857" gracePeriod=30 Oct 30 21:46:08.585446 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:08.585412 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr"] Oct 30 21:46:08.599312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:08.599280 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1915737c-vhtzr"] Oct 30 21:46:09.944047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:09.944014 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:46:09.990231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:09.990198 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:46:09.995319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:09.995288 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa12cd0-2d28-4dc9-988f-687b912375e5" path="/var/lib/kubelet/pods/6aa12cd0-2d28-4dc9-988f-687b912375e5/volumes" Oct 30 21:46:10.011418 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.011393 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-podinfo\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.011763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.011705 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-certs\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.011866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.011767 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-certs\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.011866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.011793 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-istio-token\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-istio-token\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.012020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.011905 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-credential-socket\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.012020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.011940 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-data\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.012020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.011980 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-token\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-token\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.012247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012029 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-kube-api-access-p2jnc\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-kube-api-access-p2jnc\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.012247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012068 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-socket\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.012247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012189 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-credential-socket\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.012247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012234 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-data\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.012580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012290 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-kube-api-access-qjd6b\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-kube-api-access-qjd6b\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.012580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012346 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7061e0-467e-4274-af65-b4485c807847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7061e0-467e-4274-af65-b4485c807847-istiod-ca-cert\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.012580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012404 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fdd38d-4425-45d1-a101-975dc6c631fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fdd38d-4425-45d1-a101-975dc6c631fb-istiod-ca-cert\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.012580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012500 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-socket\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.012580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012563 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-envoy\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.012883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012641 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-envoy\") pod \"37fdd38d-4425-45d1-a101-975dc6c631fb\" (UID: \"37fdd38d-4425-45d1-a101-975dc6c631fb\") " Oct 30 21:46:10.012883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.012694 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf7061e0-467e-4274-af65-b4485c807847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7061e0-467e-4274-af65-b4485c807847-istio-podinfo\") pod \"bf7061e0-467e-4274-af65-b4485c807847\" (UID: \"bf7061e0-467e-4274-af65-b4485c807847\") " Oct 30 21:46:10.014832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.014097 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-data" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.014832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.014108 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-data" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.014832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.014323 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-certs" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.014832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.014391 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-socket" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.014832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.014533 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-certs" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.014832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.014609 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-credential-socket" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.015225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.014944 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7061e0-467e-4274-af65-b4485c807847-istiod-ca-cert" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:10.015225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.015122 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-credential-socket" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.016266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.015412 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fdd38d-4425-45d1-a101-975dc6c631fb-istiod-ca-cert" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:10.016266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.016067 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-socket" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.017330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.017268 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-podinfo" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:10.018548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.018521 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:46:10.019022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.018996 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-envoy" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.019428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.019401 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-envoy" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.020771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.020656 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:46:10.020771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.020741 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-istio-token" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.020771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.020761 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-token" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.021105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.021085 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-kube-api-access-qjd6b" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "kube-api-access-qjd6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.021300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.021274 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf7061e0-467e-4274-af65-b4485c807847-istio-podinfo" pod "bf7061e0-467e-4274-af65-b4485c807847" (UID: "bf7061e0-467e-4274-af65-b4485c807847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:10.022980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.022955 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-kube-api-access-p2jnc" pod "37fdd38d-4425-45d1-a101-975dc6c631fb" (UID: "37fdd38d-4425-45d1-a101-975dc6c631fb"). InnerVolumeSpecName "kube-api-access-p2jnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.115092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115055 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-envoy\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.115261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115173 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-podinfo\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.115261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115233 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054a1029-a761-46d7-a09a-5ab39fe6f94d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a1029-a761-46d7-a09a-5ab39fe6f94d-istiod-ca-cert\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.115398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115272 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-data\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.115398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115301 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-certs\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.115398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115347 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-token\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-token\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.115398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115390 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-token\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-token\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.115658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115437 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-data\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.115658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115480 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-envoy\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.115658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115595 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-kube-api-access-fz48n\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-kube-api-access-fz48n\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.115658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115636 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-socket\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.115871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115671 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff158f4-61fe-4363-a424-7af5412ffbbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff158f4-61fe-4363-a424-7af5412ffbbf-istiod-ca-cert\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.115871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115723 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-credential-socket\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.115871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115784 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-socket\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.115871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115849 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-credential-socket\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.116238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.115955 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-certs\") pod \"054a1029-a761-46d7-a09a-5ab39fe6f94d\" (UID: \"054a1029-a761-46d7-a09a-5ab39fe6f94d\") " Oct 30 21:46:10.116238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.116008 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-certs" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.116238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.116093 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-kube-api-access-8l4mt\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-kube-api-access-8l4mt\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.116238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.116163 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-podinfo\") pod \"8ff158f4-61fe-4363-a424-7af5412ffbbf\" (UID: \"8ff158f4-61fe-4363-a424-7af5412ffbbf\") " Oct 30 21:46:10.116238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.116214 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-socket" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.116512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.116384 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-data" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117107 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-data" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117514 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054a1029-a761-46d7-a09a-5ab39fe6f94d-istiod-ca-cert" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117636 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117661 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf7061e0-467e-4274-af65-b4485c807847-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117676 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117691 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117705 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117718 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117731 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117745 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117759 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117765 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-socket" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117773 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jnc\" (UniqueName: \"kubernetes.io/projected/37fdd38d-4425-45d1-a101-975dc6c631fb-kube-api-access-p2jnc\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117812 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117827 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fdd38d-4425-45d1-a101-975dc6c631fb-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117874 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117888 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117903 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117920 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjd6b\" (UniqueName: \"kubernetes.io/projected/bf7061e0-467e-4274-af65-b4485c807847-kube-api-access-qjd6b\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117935 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf7061e0-467e-4274-af65-b4485c807847-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117948 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117962 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fdd38d-4425-45d1-a101-975dc6c631fb-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117976 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.117990 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.118039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.118004 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf7061e0-467e-4274-af65-b4485c807847-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.119405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.118302 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff158f4-61fe-4363-a424-7af5412ffbbf-istiod-ca-cert" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:10.119405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.118554 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-credential-socket" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.119405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.118632 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-podinfo" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:10.119405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.118789 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-certs" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.119405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.118978 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-credential-socket" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.120248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.120211 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-podinfo" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:10.125474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.125275 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-envoy" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.125474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.125346 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-envoy" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.125474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.125446 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-kube-api-access-8l4mt" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "kube-api-access-8l4mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.125891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.125855 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-token" pod "8ff158f4-61fe-4363-a424-7af5412ffbbf" (UID: "8ff158f4-61fe-4363-a424-7af5412ffbbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.126101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.126071 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-token" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.126672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.126639 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-kube-api-access-fz48n" pod "054a1029-a761-46d7-a09a-5ab39fe6f94d" (UID: "054a1029-a761-46d7-a09a-5ab39fe6f94d"). InnerVolumeSpecName "kube-api-access-fz48n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219803 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219833 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054a1029-a761-46d7-a09a-5ab39fe6f94d-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219843 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219851 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219859 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219869 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz48n\" (UniqueName: \"kubernetes.io/projected/054a1029-a761-46d7-a09a-5ab39fe6f94d-kube-api-access-fz48n\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219879 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ff158f4-61fe-4363-a424-7af5412ffbbf-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219887 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ff158f4-61fe-4363-a424-7af5412ffbbf-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219895 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.220406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219903 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.220406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219912 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.220406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219920 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4mt\" (UniqueName: \"kubernetes.io/projected/8ff158f4-61fe-4363-a424-7af5412ffbbf-kube-api-access-8l4mt\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.220406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219928 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ff158f4-61fe-4363-a424-7af5412ffbbf-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.220406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.219936 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054a1029-a761-46d7-a09a-5ab39fe6f94d-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.265248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.265217 4526 generic.go:356] "Generic (PLEG): container finished" podID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerID="79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec" exitCode=0 Oct 30 21:46:10.265460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.265276 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" Oct 30 21:46:10.265460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.265295 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" event={"ID":"37fdd38d-4425-45d1-a101-975dc6c631fb","Type":"ContainerDied","Data":"79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec"} Oct 30 21:46:10.265460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.265332 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2" event={"ID":"37fdd38d-4425-45d1-a101-975dc6c631fb","Type":"ContainerDied","Data":"3065877b1e484348ca57fa82825341c226979fd03c5e4f8f93b235980cff1c24"} Oct 30 21:46:10.265460 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.265354 4526 scope.go:122] "RemoveContainer" containerID="79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec" Oct 30 21:46:10.267247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.267221 4526 generic.go:356] "Generic (PLEG): container finished" podID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerID="1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857" exitCode=0 Oct 30 21:46:10.267435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.267402 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" event={"ID":"8ff158f4-61fe-4363-a424-7af5412ffbbf","Type":"ContainerDied","Data":"1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857"} Oct 30 21:46:10.267605 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.267455 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" event={"ID":"8ff158f4-61fe-4363-a424-7af5412ffbbf","Type":"ContainerDied","Data":"0bfbebbd1881846560a21ea5ce477d0240dc60d5990f57d6b990fb0ea29b2f90"} Oct 30 21:46:10.267605 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.267601 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg" Oct 30 21:46:10.272102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.269679 4526 generic.go:356] "Generic (PLEG): container finished" podID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerID="69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee" exitCode=0 Oct 30 21:46:10.272102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.269754 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" Oct 30 21:46:10.272102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.269834 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" event={"ID":"054a1029-a761-46d7-a09a-5ab39fe6f94d","Type":"ContainerDied","Data":"69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee"} Oct 30 21:46:10.272102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.269872 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7" event={"ID":"054a1029-a761-46d7-a09a-5ab39fe6f94d","Type":"ContainerDied","Data":"be9ac6e3d1d82ba153536a3145326c819dec555f81bc467587f729dad4adeacb"} Oct 30 21:46:10.278264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.278239 4526 generic.go:356] "Generic (PLEG): container finished" podID="bf7061e0-467e-4274-af65-b4485c807847" containerID="38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a" exitCode=0 Oct 30 21:46:10.278382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.278290 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" Oct 30 21:46:10.278382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.278290 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" event={"ID":"bf7061e0-467e-4274-af65-b4485c807847","Type":"ContainerDied","Data":"38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a"} Oct 30 21:46:10.278512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.278407 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl" event={"ID":"bf7061e0-467e-4274-af65-b4485c807847","Type":"ContainerDied","Data":"ac34e39f904f6c3223122b05c3b47a7f54086789b49cca4feadd20aa88c04c74"} Oct 30 21:46:10.282553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.282532 4526 scope.go:122] "RemoveContainer" containerID="79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec" Oct 30 21:46:10.283285 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:46:10.283259 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec\": container with ID starting with 79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec not found: ID does not exist" containerID="79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec" Oct 30 21:46:10.283368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.283296 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec"} err="failed to get container status \"79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec\": rpc error: code = NotFound desc = could not find container \"79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec\": container with ID starting with 79bab5cfc5560ff14f8abbc4a9d394cb1b680174b2f4d2a1077f06dad20526ec not found: ID does not exist" Oct 30 21:46:10.283368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.283320 4526 scope.go:122] "RemoveContainer" containerID="1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857" Oct 30 21:46:10.297258 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.297238 4526 scope.go:122] "RemoveContainer" containerID="1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857" Oct 30 21:46:10.298178 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:46:10.298153 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857\": container with ID starting with 1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857 not found: ID does not exist" containerID="1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857" Oct 30 21:46:10.298295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.298189 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857"} err="failed to get container status \"1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857\": rpc error: code = NotFound desc = could not find container \"1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857\": container with ID starting with 1d8031ae8d7fcb8ac0d460b18ed388917837ae18d492d935e53991fbd16a9857 not found: ID does not exist" Oct 30 21:46:10.298295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.298216 4526 scope.go:122] "RemoveContainer" containerID="69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee" Oct 30 21:46:10.310959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.310944 4526 scope.go:122] "RemoveContainer" containerID="69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee" Oct 30 21:46:10.311208 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:46:10.311187 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee\": container with ID starting with 69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee not found: ID does not exist" containerID="69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee" Oct 30 21:46:10.311255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.311215 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee"} err="failed to get container status \"69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee\": rpc error: code = NotFound desc = could not find container \"69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee\": container with ID starting with 69a61381e56a47497d90789f4c0366a968c1ca7ea3d75d8a559b00ace57cc9ee not found: ID does not exist" Oct 30 21:46:10.311255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.311231 4526 scope.go:122] "RemoveContainer" containerID="38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a" Oct 30 21:46:10.320831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.320810 4526 scope.go:122] "RemoveContainer" containerID="38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a" Oct 30 21:46:10.321120 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:46:10.321091 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a\": container with ID starting with 38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a not found: ID does not exist" containerID="38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a" Oct 30 21:46:10.321196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.321122 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a"} err="failed to get container status \"38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a\": rpc error: code = NotFound desc = could not find container \"38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a\": container with ID starting with 38a2e8d740cb4d46f33c719ccd9b7da42b27f2c26e38e1e6bc6b1cd0540c4e7a not found: ID does not exist" Oct 30 21:46:10.659619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.659584 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg"] Oct 30 21:46:10.663231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.663208 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-ljzrg"] Oct 30 21:46:10.771709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.771677 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7"] Oct 30 21:46:10.778542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.778517 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-mzxf7"] Oct 30 21:46:10.887698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.887477 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl"] Oct 30 21:46:10.891550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:10.891523 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9w4nl"] Oct 30 21:46:11.010019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:11.009982 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2"] Oct 30 21:46:11.016327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:11.016297 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-9vkz2"] Oct 30 21:46:11.988443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:11.988408 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" path="/var/lib/kubelet/pods/054a1029-a761-46d7-a09a-5ab39fe6f94d/volumes" Oct 30 21:46:11.988980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:11.988965 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" path="/var/lib/kubelet/pods/37fdd38d-4425-45d1-a101-975dc6c631fb/volumes" Oct 30 21:46:11.989410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:11.989396 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" path="/var/lib/kubelet/pods/8ff158f4-61fe-4363-a424-7af5412ffbbf/volumes" Oct 30 21:46:11.990165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:11.990150 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7061e0-467e-4274-af65-b4485c807847" path="/var/lib/kubelet/pods/bf7061e0-467e-4274-af65-b4485c807847/volumes" Oct 30 21:46:12.479420 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:46:12.479375 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:46:15.225881 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:15.225849 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:46:15.226353 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:15.226329 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:46:19.717982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:19.717945 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks"] Oct 30 21:46:19.718474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:19.718235 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" containerID="cri-o://204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a" gracePeriod=30 Oct 30 21:46:19.767466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:19.767430 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c"] Oct 30 21:46:19.767727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:19.767689 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" containerID="cri-o://0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61" gracePeriod=30 Oct 30 21:46:24.635623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.635579 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt"] Oct 30 21:46:24.636364 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636347 4526 cpu_manager.go:414] "containerMap: removing container" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" Oct 30 21:46:24.636413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636366 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" Oct 30 21:46:24.636413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636390 4526 cpu_manager.go:414] "containerMap: removing container" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" Oct 30 21:46:24.636413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636395 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" Oct 30 21:46:24.636413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636412 4526 cpu_manager.go:414] "containerMap: removing container" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" Oct 30 21:46:24.636609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636417 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" Oct 30 21:46:24.636609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636432 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" Oct 30 21:46:24.636609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636436 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" Oct 30 21:46:24.636609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636555 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fdd38d-4425-45d1-a101-975dc6c631fb" containerName="istio-proxy" Oct 30 21:46:24.636609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636568 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="054a1029-a761-46d7-a09a-5ab39fe6f94d" containerName="istio-proxy" Oct 30 21:46:24.636609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636574 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff158f4-61fe-4363-a424-7af5412ffbbf" containerName="istio-proxy" Oct 30 21:46:24.636609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.636585 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7061e0-467e-4274-af65-b4485c807847" containerName="istio-proxy" Oct 30 21:46:24.642109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.642088 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:24.694808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.694783 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-3ddec2\"" Oct 30 21:46:24.717565 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.717521 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e185477-5e52-45e4-9500-c04371ba67e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt\" (UID: \"0e185477-5e52-45e4-9500-c04371ba67e2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:24.717814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.717790 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsw5\" (UniqueName: \"kubernetes.io/projected/0e185477-5e52-45e4-9500-c04371ba67e2-kube-api-access-jrsw5\") pod \"payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt\" (UID: \"0e185477-5e52-45e4-9500-c04371ba67e2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:24.789225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.789193 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-3ddec2-dockercfg-fssxw\"" Oct 30 21:46:24.795113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.795058 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt"] Oct 30 21:46:24.819464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.819434 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsw5\" (UniqueName: \"kubernetes.io/projected/0e185477-5e52-45e4-9500-c04371ba67e2-kube-api-access-jrsw5\") pod \"payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt\" (UID: \"0e185477-5e52-45e4-9500-c04371ba67e2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:24.819720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.819700 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e185477-5e52-45e4-9500-c04371ba67e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt\" (UID: \"0e185477-5e52-45e4-9500-c04371ba67e2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:24.820391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.820369 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e185477-5e52-45e4-9500-c04371ba67e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt\" (UID: \"0e185477-5e52-45e4-9500-c04371ba67e2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:24.886718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.886690 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsw5\" (UniqueName: \"kubernetes.io/projected/0e185477-5e52-45e4-9500-c04371ba67e2-kube-api-access-jrsw5\") pod \"payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt\" (UID: \"0e185477-5e52-45e4-9500-c04371ba67e2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:24.960101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:24.959793 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:25.118609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.116657 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:46:25.203325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.203294 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:46:25.239009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.238971 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-podinfo\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.239204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.239053 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-credential-socket\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.239204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.239093 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-certs\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.239204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.239147 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-socket\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.239374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.239226 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-kube-api-access-ddvv5\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-kube-api-access-ddvv5\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.239374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.239303 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-envoy\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.239374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.239347 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500581f8-6b15-45f0-b21d-37ef93d26b53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500581f8-6b15-45f0-b21d-37ef93d26b53-istiod-ca-cert\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.240211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.239399 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-token\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-token\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.240211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.239481 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-data\") pod \"500581f8-6b15-45f0-b21d-37ef93d26b53\" (UID: \"500581f8-6b15-45f0-b21d-37ef93d26b53\") " Oct 30 21:46:25.245306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.243623 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-socket" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.245306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.244533 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.245519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.245410 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-data" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.245789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.245762 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-credential-socket" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.245890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.245781 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-certs" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.246616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.246557 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500581f8-6b15-45f0-b21d-37ef93d26b53-istiod-ca-cert" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:25.248098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.248068 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-envoy" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.251871 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.251837 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-podinfo" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:25.252015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.251848 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-token" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.253192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.253173 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-kube-api-access-ddvv5" pod "500581f8-6b15-45f0-b21d-37ef93d26b53" (UID: "500581f8-6b15-45f0-b21d-37ef93d26b53"). InnerVolumeSpecName "kube-api-access-ddvv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.346041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346007 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-socket\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.346264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346060 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-kube-api-access-l526r\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-kube-api-access-l526r\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.346264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346134 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-credential-socket\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.346264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346230 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-envoy\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.346559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346320 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-socket" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.346630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346598 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istiod-ca-cert\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.346698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346655 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-credential-socket" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.346749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346726 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-certs\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.346905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346887 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-podinfo\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.346968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.346955 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-data\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.347020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347010 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-token\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-token\") pod \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\" (UID: \"3fb216ab-5a92-4328-8538-0c7c1e2d8788\") " Oct 30 21:46:25.347278 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347254 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istiod-ca-cert" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:25.347687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347668 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500581f8-6b15-45f0-b21d-37ef93d26b53-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347690 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347703 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347716 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347729 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347744 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347758 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347770 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347783 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.347796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347796 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddvv5\" (UniqueName: \"kubernetes.io/projected/500581f8-6b15-45f0-b21d-37ef93d26b53-kube-api-access-ddvv5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.348370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347811 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500581f8-6b15-45f0-b21d-37ef93d26b53-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.348370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.347834 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-data" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.348370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.348009 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-certs" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.350046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.350005 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-podinfo" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:25.350166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.350111 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-kube-api-access-l526r" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "kube-api-access-l526r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.350690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.350662 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-envoy" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.351428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.351405 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-token" pod "3fb216ab-5a92-4328-8538-0c7c1e2d8788" (UID: "3fb216ab-5a92-4328-8538-0c7c1e2d8788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.449058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.448965 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.449058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.449004 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.449058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.449021 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.449058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.449057 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.449366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.449071 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.449366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.449086 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l526r\" (UniqueName: \"kubernetes.io/projected/3fb216ab-5a92-4328-8538-0c7c1e2d8788-kube-api-access-l526r\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.467579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.467533 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt"] Oct 30 21:46:25.469223 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:46:25.469191 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e185477_5e52_45e4_9500_c04371ba67e2.slice/crio-ff247c8b65d4c35394c1ebf67b0b161f8063eda5015e34746452bab80dc2134c WatchSource:0}: Error finding container ff247c8b65d4c35394c1ebf67b0b161f8063eda5015e34746452bab80dc2134c: Status 404 returned error can't find the container with id ff247c8b65d4c35394c1ebf67b0b161f8063eda5015e34746452bab80dc2134c Oct 30 21:46:25.521983 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.521902 4526 generic.go:356] "Generic (PLEG): container finished" podID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerID="0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61" exitCode=0 Oct 30 21:46:25.521983 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.521952 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" Oct 30 21:46:25.521983 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.521953 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" event={"ID":"3fb216ab-5a92-4328-8538-0c7c1e2d8788","Type":"ContainerDied","Data":"0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61"} Oct 30 21:46:25.522229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.522020 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c" event={"ID":"3fb216ab-5a92-4328-8538-0c7c1e2d8788","Type":"ContainerDied","Data":"e9ad351e3cb256c23930e5fc24c5ccf97716d01be875d675b2849a9d43ad6608"} Oct 30 21:46:25.522229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.522050 4526 scope.go:122] "RemoveContainer" containerID="0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61" Oct 30 21:46:25.523552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.523499 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" event={"ID":"0e185477-5e52-45e4-9500-c04371ba67e2","Type":"ContainerStarted","Data":"ff247c8b65d4c35394c1ebf67b0b161f8063eda5015e34746452bab80dc2134c"} Oct 30 21:46:25.532889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.532854 4526 generic.go:356] "Generic (PLEG): container finished" podID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerID="204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a" exitCode=0 Oct 30 21:46:25.533089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.533071 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" event={"ID":"500581f8-6b15-45f0-b21d-37ef93d26b53","Type":"ContainerDied","Data":"204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a"} Oct 30 21:46:25.533248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.533216 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" event={"ID":"500581f8-6b15-45f0-b21d-37ef93d26b53","Type":"ContainerDied","Data":"8ae6e353d92473dcad767ee979f4ee27eb34753d794fbe6c70fa74b5111a4e8f"} Oct 30 21:46:25.533307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.533245 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks" Oct 30 21:46:25.547314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.547295 4526 scope.go:122] "RemoveContainer" containerID="0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61" Oct 30 21:46:25.547637 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:46:25.547618 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61\": container with ID starting with 0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61 not found: ID does not exist" containerID="0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61" Oct 30 21:46:25.547694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.547646 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61"} err="failed to get container status \"0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61\": rpc error: code = NotFound desc = could not find container \"0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61\": container with ID starting with 0ebd109167e261c24a83176f3488240a0be3759b3542722a0fdc5e12a5d88b61 not found: ID does not exist" Oct 30 21:46:25.547694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.547664 4526 scope.go:122] "RemoveContainer" containerID="204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a" Oct 30 21:46:25.557625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.557611 4526 scope.go:122] "RemoveContainer" containerID="204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a" Oct 30 21:46:25.557906 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:46:25.557888 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a\": container with ID starting with 204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a not found: ID does not exist" containerID="204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a" Oct 30 21:46:25.557962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.557916 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a"} err="failed to get container status \"204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a\": rpc error: code = NotFound desc = could not find container \"204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a\": container with ID starting with 204e06c8b633a254ff4bb6f214e4f78f3c315793b1f8d1f8361a3a685183cb0a not found: ID does not exist" Oct 30 21:46:25.688780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.688734 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c"] Oct 30 21:46:25.692653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.692614 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-gps5c"] Oct 30 21:46:25.802199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.802162 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks"] Oct 30 21:46:25.809996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.809966 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-9zvks"] Oct 30 21:46:25.987943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.987909 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" path="/var/lib/kubelet/pods/3fb216ab-5a92-4328-8538-0c7c1e2d8788/volumes" Oct 30 21:46:25.988691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:25.988672 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" path="/var/lib/kubelet/pods/500581f8-6b15-45f0-b21d-37ef93d26b53/volumes" Oct 30 21:46:26.543583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.543537 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" event={"ID":"0e185477-5e52-45e4-9500-c04371ba67e2","Type":"ContainerStarted","Data":"17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4"} Oct 30 21:46:26.543791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.543622 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:26.660249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.660197 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" podStartSLOduration=2.660182427 podStartE2EDuration="2.660182427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:26.657784454 +0000 UTC m=+1776.791157652" watchObservedRunningTime="2026-10-30 21:46:26.660182427 +0000 UTC m=+1776.793555625" Oct 30 21:46:26.805625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.805539 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552"] Oct 30 21:46:26.806122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.805822 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" containerID="cri-o://9f392e41777bec010e21aa99057cb85592f57e64e06d7f17389b943a57396dce" gracePeriod=30 Oct 30 21:46:26.810349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.810325 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4"] Oct 30 21:46:26.810576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.810552 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" containerID="cri-o://45802f25775ee604396d4c925d78ade4b44befbb260ae9e0a871e6282568e920" gracePeriod=30 Oct 30 21:46:26.816057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.816032 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb"] Oct 30 21:46:26.816282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.816241 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" containerID="cri-o://b69daf3dd4f1f486c38f0888e90c4962cbf4ee3a0983077a7f0b9e77b1709b81" gracePeriod=30 Oct 30 21:46:26.874141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.874104 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f"] Oct 30 21:46:26.875753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:26.874364 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" containerID="cri-o://87d43f7443f708813824a56a0113174b6a5b581b76eb8c41638af187fa8ae5e8" gracePeriod=30 Oct 30 21:46:29.154361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.154320 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:29.154890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.154374 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:29.374746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.374705 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc"] Oct 30 21:46:29.375614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.375593 4526 cpu_manager.go:414] "containerMap: removing container" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" Oct 30 21:46:29.375614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.375616 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" Oct 30 21:46:29.375813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.375670 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" Oct 30 21:46:29.375813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.375681 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" Oct 30 21:46:29.375916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.375848 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="500581f8-6b15-45f0-b21d-37ef93d26b53" containerName="istio-proxy" Oct 30 21:46:29.375916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.375871 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb216ab-5a92-4328-8538-0c7c1e2d8788" containerName="istio-proxy" Oct 30 21:46:29.383580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.381263 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:29.438234 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.438144 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc"] Oct 30 21:46:29.481386 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.481358 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fbb72e\"" Oct 30 21:46:29.497148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.497120 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fbb72e-dockercfg-6mgzj\"" Oct 30 21:46:29.534908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.534873 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv"] Oct 30 21:46:29.539319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.539303 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:29.557408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.557382 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvjj\" (UniqueName: \"kubernetes.io/projected/975eb4b3-04cd-48f5-97ff-c888f0502f39-kube-api-access-mmvjj\") pod \"payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc\" (UID: \"975eb4b3-04cd-48f5-97ff-c888f0502f39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:29.557579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.557547 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975eb4b3-04cd-48f5-97ff-c888f0502f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc\" (UID: \"975eb4b3-04cd-48f5-97ff-c888f0502f39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:29.573945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.573912 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv"] Oct 30 21:46:29.660838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.660794 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv\" (UID: \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:29.661060 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.660859 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvjj\" (UniqueName: \"kubernetes.io/projected/975eb4b3-04cd-48f5-97ff-c888f0502f39-kube-api-access-mmvjj\") pod \"payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc\" (UID: \"975eb4b3-04cd-48f5-97ff-c888f0502f39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:29.661060 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.660900 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-kube-api-access-bgldj\") pod \"payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv\" (UID: \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:29.661060 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.661011 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975eb4b3-04cd-48f5-97ff-c888f0502f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc\" (UID: \"975eb4b3-04cd-48f5-97ff-c888f0502f39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:29.662050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.662014 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975eb4b3-04cd-48f5-97ff-c888f0502f39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc\" (UID: \"975eb4b3-04cd-48f5-97ff-c888f0502f39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:29.720230 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.720158 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvjj\" (UniqueName: \"kubernetes.io/projected/975eb4b3-04cd-48f5-97ff-c888f0502f39-kube-api-access-mmvjj\") pod \"payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc\" (UID: \"975eb4b3-04cd-48f5-97ff-c888f0502f39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:29.762346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.762294 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv\" (UID: \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:29.762606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.762357 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-kube-api-access-bgldj\") pod \"payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv\" (UID: \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:29.763124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.763104 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv\" (UID: \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:29.835592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.829768 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-kube-api-access-bgldj\") pod \"payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv\" (UID: \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:29.854015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:29.853980 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:30.011901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.011806 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:30.265449 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:46:30.265406 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2f44f4_ec6c_44b1_b0c9_5888abe8fe6d.slice/crio-3b826ba6c3460de257e38cf1507b70898c33181f4a7614e83d6c4ba8f5eeec0d WatchSource:0}: Error finding container 3b826ba6c3460de257e38cf1507b70898c33181f4a7614e83d6c4ba8f5eeec0d: Status 404 returned error can't find the container with id 3b826ba6c3460de257e38cf1507b70898c33181f4a7614e83d6c4ba8f5eeec0d Oct 30 21:46:30.266745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.266644 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv"] Oct 30 21:46:30.426745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.426718 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc"] Oct 30 21:46:30.430020 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:46:30.429980 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975eb4b3_04cd_48f5_97ff_c888f0502f39.slice/crio-42ec6c2fb7628f8f3c70eeb5eabff07254934ee796ab20dde57e72912eb645b0 WatchSource:0}: Error finding container 42ec6c2fb7628f8f3c70eeb5eabff07254934ee796ab20dde57e72912eb645b0: Status 404 returned error can't find the container with id 42ec6c2fb7628f8f3c70eeb5eabff07254934ee796ab20dde57e72912eb645b0 Oct 30 21:46:30.581182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.581138 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" event={"ID":"975eb4b3-04cd-48f5-97ff-c888f0502f39","Type":"ContainerStarted","Data":"99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077"} Oct 30 21:46:30.581400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.581215 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" event={"ID":"975eb4b3-04cd-48f5-97ff-c888f0502f39","Type":"ContainerStarted","Data":"42ec6c2fb7628f8f3c70eeb5eabff07254934ee796ab20dde57e72912eb645b0"} Oct 30 21:46:30.581400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.581271 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:30.586110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.586077 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" event={"ID":"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d","Type":"ContainerStarted","Data":"06f8c4b1feaaee63ae844791bb104b92eb4be0257de0663f3ad723d08d4e080d"} Oct 30 21:46:30.586308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.586137 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" event={"ID":"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d","Type":"ContainerStarted","Data":"3b826ba6c3460de257e38cf1507b70898c33181f4a7614e83d6c4ba8f5eeec0d"} Oct 30 21:46:30.587006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.586982 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:30.717294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.717243 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" podStartSLOduration=1.717229373 podStartE2EDuration="1.717229373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:30.714170616 +0000 UTC m=+1780.847543814" watchObservedRunningTime="2026-10-30 21:46:30.717229373 +0000 UTC m=+1780.850602632" Oct 30 21:46:30.839282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:30.839220 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" podStartSLOduration=1.839197563 podStartE2EDuration="1.839197563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:30.832735082 +0000 UTC m=+1780.966108303" watchObservedRunningTime="2026-10-30 21:46:30.839197563 +0000 UTC m=+1780.972570762" Oct 30 21:46:31.270578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:31.270277 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:31.270578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:31.270338 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:31.753527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:31.752676 4526 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:31.753527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:31.752742 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:32.627978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.625913 4526 generic.go:356] "Generic (PLEG): container finished" podID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerID="b69daf3dd4f1f486c38f0888e90c4962cbf4ee3a0983077a7f0b9e77b1709b81" exitCode=0 Oct 30 21:46:32.627978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.626121 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" event={"ID":"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076","Type":"ContainerDied","Data":"b69daf3dd4f1f486c38f0888e90c4962cbf4ee3a0983077a7f0b9e77b1709b81"} Oct 30 21:46:32.633332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.633090 4526 generic.go:356] "Generic (PLEG): container finished" podID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerID="45802f25775ee604396d4c925d78ade4b44befbb260ae9e0a871e6282568e920" exitCode=0 Oct 30 21:46:32.633332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.633257 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" event={"ID":"20ee9708-4119-4fcd-b9b2-76943d9d3c8b","Type":"ContainerDied","Data":"45802f25775ee604396d4c925d78ade4b44befbb260ae9e0a871e6282568e920"} Oct 30 21:46:32.636083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.635914 4526 generic.go:356] "Generic (PLEG): container finished" podID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerID="87d43f7443f708813824a56a0113174b6a5b581b76eb8c41638af187fa8ae5e8" exitCode=0 Oct 30 21:46:32.636083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.636018 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" event={"ID":"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c","Type":"ContainerDied","Data":"87d43f7443f708813824a56a0113174b6a5b581b76eb8c41638af187fa8ae5e8"} Oct 30 21:46:32.639841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.638774 4526 generic.go:356] "Generic (PLEG): container finished" podID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerID="9f392e41777bec010e21aa99057cb85592f57e64e06d7f17389b943a57396dce" exitCode=0 Oct 30 21:46:32.639841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.639770 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" event={"ID":"bad8e78e-91e7-49d4-8078-406f90fb35e4","Type":"ContainerDied","Data":"9f392e41777bec010e21aa99057cb85592f57e64e06d7f17389b943a57396dce"} Oct 30 21:46:32.729529 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.728630 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:46:32.804553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.803836 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:46:32.809101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.809077 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:46:32.815638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.815616 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:46:32.899322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899231 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad8e78e-91e7-49d4-8078-406f90fb35e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e78e-91e7-49d4-8078-406f90fb35e4-istiod-ca-cert\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.899322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899304 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istiod-ca-cert\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.899613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899338 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istiod-ca-cert\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.899613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899380 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-podinfo\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.899613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899413 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-kube-api-access-zz42n\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-kube-api-access-zz42n\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.899613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899458 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-token\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-token\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.899613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899514 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-kube-api-access-52sb8\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-kube-api-access-52sb8\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.899613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899546 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-podinfo\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.899613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899581 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-envoy\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899629 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-token\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-token\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899659 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-credential-socket\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899711 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istiod-ca-cert" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899718 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-credential-socket\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899775 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istiod-ca-cert" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899802 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-envoy\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899835 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-socket\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899954 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-credential-socket" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.899992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.899866 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-kube-api-access-rhhg5\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-kube-api-access-rhhg5\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900023 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-socket\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900062 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-data\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900089 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-token\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-token\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900131 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-certs\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900162 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istiod-ca-cert\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900196 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-data\") pod \"bad8e78e-91e7-49d4-8078-406f90fb35e4\" (UID: \"bad8e78e-91e7-49d4-8078-406f90fb35e4\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900241 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-envoy\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900269 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-socket\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900383 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-token\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-token\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900393 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad8e78e-91e7-49d4-8078-406f90fb35e4-istiod-ca-cert" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900420 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-certs\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.900510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900472 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-podinfo\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900619 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-credential-socket" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900663 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-credential-socket\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900738 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-certs\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900769 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-certs\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900850 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-data\") pod \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\" (UID: \"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900915 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-envoy\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.900978 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-socket\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.901011 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-credential-socket\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.901149 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-kube-api-access-lxs6j\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-kube-api-access-lxs6j\") pod \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\" (UID: \"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.901199 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-podinfo\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.901320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.901232 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-data\") pod \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\" (UID: \"20ee9708-4119-4fcd-b9b2-76943d9d3c8b\") " Oct 30 21:46:32.903160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.902849 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e78e-91e7-49d4-8078-406f90fb35e4-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:32.903160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.902873 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:32.903160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.902887 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:32.903160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.902902 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:32.903160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.902917 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:32.903160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.903113 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-data" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.907424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.907346 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-certs" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.909228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.909200 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-socket" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.909331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.909263 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-data" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.909840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.909813 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-envoy" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.910063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.910039 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-socket" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.910320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.910297 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-credential-socket" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.911336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.911100 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-kube-api-access-52sb8" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "kube-api-access-52sb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:32.911336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.911216 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-kube-api-access-zz42n" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "kube-api-access-zz42n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:32.911336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.911296 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-podinfo" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:32.911783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.911452 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-data" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.913096 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.913069 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-socket" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.913220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.913201 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-kube-api-access-rhhg5" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "kube-api-access-rhhg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:32.913291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.913270 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-envoy" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.913809 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.913727 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-token" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:32.913809 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.913793 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-envoy" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.914009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.913922 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-certs" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.914074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.914045 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-credential-socket" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.914178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.914146 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istiod-ca-cert" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:32.914252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.914175 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-envoy" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.914252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.914191 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-certs" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.914557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.914357 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-token" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:32.914557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.914453 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-socket" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.914711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.914607 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-data" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.914711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.914643 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-certs" pod "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" (UID: "f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:32.915450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.915411 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-podinfo" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:32.915992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.915958 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-kube-api-access-lxs6j" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "kube-api-access-lxs6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:32.916201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.916164 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-token" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:32.916530 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.916506 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-token" pod "bad8e78e-91e7-49d4-8078-406f90fb35e4" (UID: "bad8e78e-91e7-49d4-8078-406f90fb35e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:32.916711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.916688 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-podinfo" pod "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" (UID: "e4ec57f7-0478-4d5c-8189-a4e4c6168a2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:32.916790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:32.916722 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-podinfo" pod "20ee9708-4119-4fcd-b9b2-76943d9d3c8b" (UID: "20ee9708-4119-4fcd-b9b2-76943d9d3c8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:33.004373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004335 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004369 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004381 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004390 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004401 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004409 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004418 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004427 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004437 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004445 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004454 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004462 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004470 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxs6j\" (UniqueName: \"kubernetes.io/projected/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-kube-api-access-lxs6j\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004478 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004499 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004509 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004518 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz42n\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-kube-api-access-zz42n\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004526 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004535 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52sb8\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-kube-api-access-52sb8\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004543 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004551 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004561 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004569 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004579 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004587 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhg5\" (UniqueName: \"kubernetes.io/projected/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076-kube-api-access-rhhg5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004597 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ee9708-4119-4fcd-b9b2-76943d9d3c8b-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004605 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004613 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004621 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004639 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.004685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.004649 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad8e78e-91e7-49d4-8078-406f90fb35e4-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:33.648069 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.648036 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" Oct 30 21:46:33.648642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.648066 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4" event={"ID":"20ee9708-4119-4fcd-b9b2-76943d9d3c8b","Type":"ContainerDied","Data":"c6426ebc91f942cdea6728f91e5b15af6ca16eed6b67718a88c8946983babcd1"} Oct 30 21:46:33.648642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.648126 4526 scope.go:122] "RemoveContainer" containerID="45802f25775ee604396d4c925d78ade4b44befbb260ae9e0a871e6282568e920" Oct 30 21:46:33.650049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.650024 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" Oct 30 21:46:33.650170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.650066 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f" event={"ID":"e4ec57f7-0478-4d5c-8189-a4e4c6168a2c","Type":"ContainerDied","Data":"31a6ab997224eebc3b8e12d0e92b9b2f7107ac8d300e1c8ac87bcb6bf870f68e"} Oct 30 21:46:33.658290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.658263 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" Oct 30 21:46:33.658290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.658282 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552" event={"ID":"bad8e78e-91e7-49d4-8078-406f90fb35e4","Type":"ContainerDied","Data":"f7a9edeb04dfab67f2db3713c93c441574f04e4cba83b3d58d4c488295aecd9f"} Oct 30 21:46:33.660481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.660420 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" event={"ID":"f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076","Type":"ContainerDied","Data":"0f4462bc15f96623137f672889b3a338379a67e6bf66c118d20dd456009c77d8"} Oct 30 21:46:33.660481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.660433 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb" Oct 30 21:46:33.668550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.667248 4526 scope.go:122] "RemoveContainer" containerID="87d43f7443f708813824a56a0113174b6a5b581b76eb8c41638af187fa8ae5e8" Oct 30 21:46:33.677640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.677618 4526 scope.go:122] "RemoveContainer" containerID="9f392e41777bec010e21aa99057cb85592f57e64e06d7f17389b943a57396dce" Oct 30 21:46:33.688787 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.688762 4526 scope.go:122] "RemoveContainer" containerID="b69daf3dd4f1f486c38f0888e90c4962cbf4ee3a0983077a7f0b9e77b1709b81" Oct 30 21:46:33.814430 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.814400 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4"] Oct 30 21:46:33.863400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.863370 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-hrjp4"] Oct 30 21:46:33.928907 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.928817 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb"] Oct 30 21:46:33.932322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.932297 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-q57mb"] Oct 30 21:46:33.987371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.987343 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" path="/var/lib/kubelet/pods/20ee9708-4119-4fcd-b9b2-76943d9d3c8b/volumes" Oct 30 21:46:33.987962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:33.987945 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" path="/var/lib/kubelet/pods/f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076/volumes" Oct 30 21:46:34.040672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:34.040628 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552"] Oct 30 21:46:34.047599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:34.047571 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tq552"] Oct 30 21:46:34.155904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:34.155867 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f"] Oct 30 21:46:34.160715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:34.160687 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-4wb4f"] Oct 30 21:46:35.987212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:35.987179 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" path="/var/lib/kubelet/pods/bad8e78e-91e7-49d4-8078-406f90fb35e4/volumes" Oct 30 21:46:35.987684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:35.987668 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" path="/var/lib/kubelet/pods/e4ec57f7-0478-4d5c-8189-a4e4c6168a2c/volumes" Oct 30 21:46:37.554641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:37.554610 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:46:41.598537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:41.598469 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:46:42.480092 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:46:42.480054 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:46:42.642855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:42.642827 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:46:51.340652 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:51.340622 4526 scope.go:122] "RemoveContainer" containerID="bf9051bc35f51af9b332de093307661c182757a78d7d2dbfc2b425391fc9a467" Oct 30 21:46:51.350997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:51.350976 4526 scope.go:122] "RemoveContainer" containerID="c4fd8cce7ff54a5c0cdc857a35c15933f10dce0ffe62079659d4f47bcc8f645a" Oct 30 21:46:51.365032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:51.365002 4526 scope.go:122] "RemoveContainer" containerID="2efc8b92cc8c6c9bb2dd5869f7b4a6b974fad9b252218ca67eee1533e0073939" Oct 30 21:46:51.375330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:51.375309 4526 scope.go:122] "RemoveContainer" containerID="5faca56c00f6f1063238fea6620a22ca2ba12d9c0705a9deeed4573e7e3c4643" Oct 30 21:46:51.386032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:51.386013 4526 scope.go:122] "RemoveContainer" containerID="aa3376ec79b5f47cdb73a2a92a5d0eca839962b9dfe1004f9e6342f487801542" Oct 30 21:46:51.395634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:46:51.395616 4526 scope.go:122] "RemoveContainer" containerID="298bc9eca67cc95342a7696b8625fa6141db4c71f859151aeeb04fe37032776a" Oct 30 21:47:09.680424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.680387 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7289-f209e6"] Oct 30 21:47:09.681104 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681089 4526 cpu_manager.go:414] "containerMap: removing container" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" Oct 30 21:47:09.681162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681106 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" Oct 30 21:47:09.681162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681133 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" Oct 30 21:47:09.681162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681138 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" Oct 30 21:47:09.681261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681165 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" Oct 30 21:47:09.681261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681170 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" Oct 30 21:47:09.681261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681181 4526 cpu_manager.go:414] "containerMap: removing container" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" Oct 30 21:47:09.681261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681187 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" Oct 30 21:47:09.681384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681268 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad8e78e-91e7-49d4-8078-406f90fb35e4" containerName="istio-proxy" Oct 30 21:47:09.681384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681280 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d3c749-91d6-4ab8-b1d3-5a6f27c1b076" containerName="istio-proxy" Oct 30 21:47:09.681384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681290 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ec57f7-0478-4d5c-8189-a4e4c6168a2c" containerName="istio-proxy" Oct 30 21:47:09.681384 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.681298 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ee9708-4119-4fcd-b9b2-76943d9d3c8b" containerName="istio-proxy" Oct 30 21:47:09.687969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.687951 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:09.736411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.736377 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-f209e6"] Oct 30 21:47:09.780809 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.780778 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m2hp5\"" Oct 30 21:47:09.816949 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.816912 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qwdz\" (UniqueName: \"kubernetes.io/projected/65f0c4b7-de5a-41a6-b210-cf9bdc36dce8-kube-api-access-7qwdz\") pod \"test-curl-7289-f209e6\" (UID: \"65f0c4b7-de5a-41a6-b210-cf9bdc36dce8\") " pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:09.918710 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.918676 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qwdz\" (UniqueName: \"kubernetes.io/projected/65f0c4b7-de5a-41a6-b210-cf9bdc36dce8-kube-api-access-7qwdz\") pod \"test-curl-7289-f209e6\" (UID: \"65f0c4b7-de5a-41a6-b210-cf9bdc36dce8\") " pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:09.976599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:09.976526 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qwdz\" (UniqueName: \"kubernetes.io/projected/65f0c4b7-de5a-41a6-b210-cf9bdc36dce8-kube-api-access-7qwdz\") pod \"test-curl-7289-f209e6\" (UID: \"65f0c4b7-de5a-41a6-b210-cf9bdc36dce8\") " pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:10.000526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:10.000476 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:10.380536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:10.380504 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-f209e6"] Oct 30 21:47:10.382121 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:10.382094 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65f0c4b7_de5a_41a6_b210_cf9bdc36dce8.slice/crio-389f2ac1c4f6ac3fa57ebf9b5c5ad46c7dc4f89a5a71049b3b9eef9c236c0f95 WatchSource:0}: Error finding container 389f2ac1c4f6ac3fa57ebf9b5c5ad46c7dc4f89a5a71049b3b9eef9c236c0f95: Status 404 returned error can't find the container with id 389f2ac1c4f6ac3fa57ebf9b5c5ad46c7dc4f89a5a71049b3b9eef9c236c0f95 Oct 30 21:47:10.384289 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:10.384270 4526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:47:10.967520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:10.967455 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-f209e6" event={"ID":"65f0c4b7-de5a-41a6-b210-cf9bdc36dce8","Type":"ContainerStarted","Data":"389f2ac1c4f6ac3fa57ebf9b5c5ad46c7dc4f89a5a71049b3b9eef9c236c0f95"} Oct 30 21:47:12.052085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:12.052047 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz"] Oct 30 21:47:12.101697 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:12.101659 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d81bd7-6dxfz"] Oct 30 21:47:12.484863 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:12.484754 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:47:13.987162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:13.987129 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1cfe939-1537-4af8-b67a-c9d701566b33" path="/var/lib/kubelet/pods/a1cfe939-1537-4af8-b67a-c9d701566b33/volumes" Oct 30 21:47:13.995832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:13.995796 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-f209e6" event={"ID":"65f0c4b7-de5a-41a6-b210-cf9bdc36dce8","Type":"ContainerStarted","Data":"293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c"} Oct 30 21:47:14.111667 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:14.111610 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7289-f209e6" podStartSLOduration=2.370955549 podStartE2EDuration="5.111595148s" totalImagesPullingTime="2.740639599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:14.10768719 +0000 UTC m=+1824.241060388" watchObservedRunningTime="2026-10-30 21:47:14.111595148 +0000 UTC m=+1824.244968346" Oct 30 21:47:15.552709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.552669 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7289-f209e6"] Oct 30 21:47:15.556540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.556513 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7289-f209e6"] Oct 30 21:47:15.704762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.704726 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7289-790a42"] Oct 30 21:47:15.705408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.705393 4526 cpu_manager.go:414] "containerMap: removing container" podUID="65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" containerName="test-curl-7289-f209e6" Oct 30 21:47:15.705465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.705409 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" containerName="test-curl-7289-f209e6" Oct 30 21:47:15.705548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.705538 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" containerName="test-curl-7289-f209e6" Oct 30 21:47:15.709332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.709310 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:15.766999 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.766965 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-790a42"] Oct 30 21:47:15.806557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.806457 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fckzk\" (UniqueName: \"kubernetes.io/projected/19394ad1-d4c8-4d0c-9062-1a939c72b7cf-kube-api-access-fckzk\") pod \"test-curl-7289-790a42\" (UID: \"19394ad1-d4c8-4d0c-9062-1a939c72b7cf\") " pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:15.907411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.907373 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fckzk\" (UniqueName: \"kubernetes.io/projected/19394ad1-d4c8-4d0c-9062-1a939c72b7cf-kube-api-access-fckzk\") pod \"test-curl-7289-790a42\" (UID: \"19394ad1-d4c8-4d0c-9062-1a939c72b7cf\") " pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:15.964137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:15.964110 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fckzk\" (UniqueName: \"kubernetes.io/projected/19394ad1-d4c8-4d0c-9062-1a939c72b7cf-kube-api-access-fckzk\") pod \"test-curl-7289-790a42\" (UID: \"19394ad1-d4c8-4d0c-9062-1a939c72b7cf\") " pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:16.010713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:16.010655 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7289-f209e6" podUID="65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" containerName="test-curl-7289-f209e6" containerID="cri-o://293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c" gracePeriod=2 Oct 30 21:47:16.022842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:16.022811 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:16.400122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:16.400087 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-790a42"] Oct 30 21:47:16.401684 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:16.401651 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19394ad1_d4c8_4d0c_9062_1a939c72b7cf.slice/crio-ea6d5368c24ab0f5f66a2f8ebf468837c93ba269e08cfd7b1bbde59b497b6251 WatchSource:0}: Error finding container ea6d5368c24ab0f5f66a2f8ebf468837c93ba269e08cfd7b1bbde59b497b6251: Status 404 returned error can't find the container with id ea6d5368c24ab0f5f66a2f8ebf468837c93ba269e08cfd7b1bbde59b497b6251 Oct 30 21:47:17.020389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:17.020346 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-790a42" event={"ID":"19394ad1-d4c8-4d0c-9062-1a939c72b7cf","Type":"ContainerStarted","Data":"9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef"} Oct 30 21:47:17.020389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:17.020397 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-790a42" event={"ID":"19394ad1-d4c8-4d0c-9062-1a939c72b7cf","Type":"ContainerStarted","Data":"ea6d5368c24ab0f5f66a2f8ebf468837c93ba269e08cfd7b1bbde59b497b6251"} Oct 30 21:47:17.140168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:17.140115 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7289-790a42" podStartSLOduration=2.140099427 podStartE2EDuration="2.140099427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:17.13904149 +0000 UTC m=+1827.272414688" watchObservedRunningTime="2026-10-30 21:47:17.140099427 +0000 UTC m=+1827.273472625" Oct 30 21:47:18.263661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.263628 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:18.338218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.338189 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f0c4b7-de5a-41a6-b210-cf9bdc36dce8-kube-api-access-7qwdz\" (UniqueName: \"kubernetes.io/projected/65f0c4b7-de5a-41a6-b210-cf9bdc36dce8-kube-api-access-7qwdz\") pod \"65f0c4b7-de5a-41a6-b210-cf9bdc36dce8\" (UID: \"65f0c4b7-de5a-41a6-b210-cf9bdc36dce8\") " Oct 30 21:47:18.340501 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.340450 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f0c4b7-de5a-41a6-b210-cf9bdc36dce8-kube-api-access-7qwdz" pod "65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" (UID: "65f0c4b7-de5a-41a6-b210-cf9bdc36dce8"). InnerVolumeSpecName "kube-api-access-7qwdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:18.440091 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.440056 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qwdz\" (UniqueName: \"kubernetes.io/projected/65f0c4b7-de5a-41a6-b210-cf9bdc36dce8-kube-api-access-7qwdz\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:18.716946 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.716864 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7289-790a42"] Oct 30 21:47:18.722006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.721967 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7289-790a42"] Oct 30 21:47:18.907375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.907345 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7289-9c3aef"] Oct 30 21:47:18.908093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.908076 4526 cpu_manager.go:414] "containerMap: removing container" podUID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" containerName="test-curl-7289-790a42" Oct 30 21:47:18.908142 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.908097 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" containerName="test-curl-7289-790a42" Oct 30 21:47:18.908220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.908210 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" containerName="test-curl-7289-790a42" Oct 30 21:47:18.911704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.911689 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:18.946951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.946913 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2vmq\" (UniqueName: \"kubernetes.io/projected/114730a4-766c-4a27-ba62-ae142a65d355-kube-api-access-q2vmq\") pod \"test-curl-7289-9c3aef\" (UID: \"114730a4-766c-4a27-ba62-ae142a65d355\") " pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:18.968339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:18.968261 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-9c3aef"] Oct 30 21:47:19.035472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.035441 4526 generic.go:356] "Generic (PLEG): container finished" podID="65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" containerID="293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c" exitCode=137 Oct 30 21:47:19.035472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.035502 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:19.035727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.035535 4526 scope.go:122] "RemoveContainer" containerID="293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c" Oct 30 21:47:19.035727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.035679 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7289-790a42" podUID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" containerName="test-curl-7289-790a42" containerID="cri-o://9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef" gracePeriod=2 Oct 30 21:47:19.046482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.046458 4526 scope.go:122] "RemoveContainer" containerID="293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c" Oct 30 21:47:19.046790 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:19.046767 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c\": container with ID starting with 293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c not found: ID does not exist" containerID="293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c" Oct 30 21:47:19.046841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.046799 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c"} err="failed to get container status \"293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c\": rpc error: code = NotFound desc = could not find container \"293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c\": container with ID starting with 293b6a68148fa133d29e052636b098a0491cdd6f80a7a03d6d74ed70e709a43c not found: ID does not exist" Oct 30 21:47:19.048570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.048549 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2vmq\" (UniqueName: \"kubernetes.io/projected/114730a4-766c-4a27-ba62-ae142a65d355-kube-api-access-q2vmq\") pod \"test-curl-7289-9c3aef\" (UID: \"114730a4-766c-4a27-ba62-ae142a65d355\") " pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:19.086469 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:19.086427 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-790a42\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:19.107337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.107305 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2vmq\" (UniqueName: \"kubernetes.io/projected/114730a4-766c-4a27-ba62-ae142a65d355-kube-api-access-q2vmq\") pod \"test-curl-7289-9c3aef\" (UID: \"114730a4-766c-4a27-ba62-ae142a65d355\") " pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:19.137043 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:19.137005 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-790a42\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:19.188271 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:19.188230 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-f209e6\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:19.224943 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.224869 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:19.623162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.623127 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-9c3aef"] Oct 30 21:47:19.624775 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:19.624746 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod114730a4_766c_4a27_ba62_ae142a65d355.slice/crio-2742a15ddfca65f33890c2996a2ff9aff35dacad4155cf4dc1603c9ce1cf956a WatchSource:0}: Error finding container 2742a15ddfca65f33890c2996a2ff9aff35dacad4155cf4dc1603c9ce1cf956a: Status 404 returned error can't find the container with id 2742a15ddfca65f33890c2996a2ff9aff35dacad4155cf4dc1603c9ce1cf956a Oct 30 21:47:19.995678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:19.995600 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" path="/var/lib/kubelet/pods/65f0c4b7-de5a-41a6-b210-cf9bdc36dce8/volumes" Oct 30 21:47:20.040460 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:20.040413 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-790a42\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:20.045198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:20.045164 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-9c3aef" event={"ID":"114730a4-766c-4a27-ba62-ae142a65d355","Type":"ContainerStarted","Data":"c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58"} Oct 30 21:47:20.045198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:20.045205 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-9c3aef" event={"ID":"114730a4-766c-4a27-ba62-ae142a65d355","Type":"ContainerStarted","Data":"2742a15ddfca65f33890c2996a2ff9aff35dacad4155cf4dc1603c9ce1cf956a"} Oct 30 21:47:20.091167 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:20.091126 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-f209e6\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="65f0c4b7-de5a-41a6-b210-cf9bdc36dce8" pod="opendatahub/test-curl-7289-f209e6" Oct 30 21:47:20.203908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:20.203846 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7289-9c3aef" podStartSLOduration=2.203829262 podStartE2EDuration="2.203829262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:20.202957612 +0000 UTC m=+1830.336330811" watchObservedRunningTime="2026-10-30 21:47:20.203829262 +0000 UTC m=+1830.337202464" Oct 30 21:47:21.313118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.313050 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:21.381427 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.381391 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19394ad1-d4c8-4d0c-9062-1a939c72b7cf-kube-api-access-fckzk\" (UniqueName: \"kubernetes.io/projected/19394ad1-d4c8-4d0c-9062-1a939c72b7cf-kube-api-access-fckzk\") pod \"19394ad1-d4c8-4d0c-9062-1a939c72b7cf\" (UID: \"19394ad1-d4c8-4d0c-9062-1a939c72b7cf\") " Oct 30 21:47:21.383822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.383786 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19394ad1-d4c8-4d0c-9062-1a939c72b7cf-kube-api-access-fckzk" pod "19394ad1-d4c8-4d0c-9062-1a939c72b7cf" (UID: "19394ad1-d4c8-4d0c-9062-1a939c72b7cf"). InnerVolumeSpecName "kube-api-access-fckzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:21.483865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.483821 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fckzk\" (UniqueName: \"kubernetes.io/projected/19394ad1-d4c8-4d0c-9062-1a939c72b7cf-kube-api-access-fckzk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:21.569836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.569753 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb"] Oct 30 21:47:21.574150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.574132 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.600502 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.600462 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7289-9c3aef"] Oct 30 21:47:21.603836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.603814 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7289-9c3aef"] Oct 30 21:47:21.633045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.633008 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb"] Oct 30 21:47:21.646264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.646225 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:47:21.646264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.646275 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:47:21.649773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.649743 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:21.649936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.649847 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:47:21.686833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.686795 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kzx\" (UniqueName: \"kubernetes.io/projected/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-kube-api-access-k8kzx\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.686988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.686861 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.687061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.686976 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.744743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.744706 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7289-3a6fb6"] Oct 30 21:47:21.745410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.745396 4526 cpu_manager.go:414] "containerMap: removing container" podUID="114730a4-766c-4a27-ba62-ae142a65d355" containerName="test-curl-7289-9c3aef" Oct 30 21:47:21.745468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.745412 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114730a4-766c-4a27-ba62-ae142a65d355" containerName="test-curl-7289-9c3aef" Oct 30 21:47:21.745559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.745550 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="114730a4-766c-4a27-ba62-ae142a65d355" containerName="test-curl-7289-9c3aef" Oct 30 21:47:21.749415 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.749399 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:21.787979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.787946 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.788148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.788018 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.788148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.788089 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jg9s\" (UniqueName: \"kubernetes.io/projected/52429b8f-fff8-4b23-a5e5-ae644cd8333e-kube-api-access-7jg9s\") pod \"test-curl-7289-3a6fb6\" (UID: \"52429b8f-fff8-4b23-a5e5-ae644cd8333e\") " pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:21.788429 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.788410 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kzx\" (UniqueName: \"kubernetes.io/projected/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-kube-api-access-k8kzx\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.788854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.788785 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.790753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.790730 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.804415 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.804389 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-3a6fb6"] Oct 30 21:47:21.846009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.845928 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kzx\" (UniqueName: \"kubernetes.io/projected/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-kube-api-access-k8kzx\") pod \"maas-api-revoke-sub-49c3b3a2-wcgnb\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.888934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.888897 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:21.889467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.889444 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jg9s\" (UniqueName: \"kubernetes.io/projected/52429b8f-fff8-4b23-a5e5-ae644cd8333e-kube-api-access-7jg9s\") pod \"test-curl-7289-3a6fb6\" (UID: \"52429b8f-fff8-4b23-a5e5-ae644cd8333e\") " pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:21.946812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.946779 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jg9s\" (UniqueName: \"kubernetes.io/projected/52429b8f-fff8-4b23-a5e5-ae644cd8333e-kube-api-access-7jg9s\") pod \"test-curl-7289-3a6fb6\" (UID: \"52429b8f-fff8-4b23-a5e5-ae644cd8333e\") " pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:21.988148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:21.988111 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" path="/var/lib/kubelet/pods/19394ad1-d4c8-4d0c-9062-1a939c72b7cf/volumes" Oct 30 21:47:22.062064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.062039 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:22.062315 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.062288 4526 generic.go:356] "Generic (PLEG): container finished" podID="19394ad1-d4c8-4d0c-9062-1a939c72b7cf" containerID="9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef" exitCode=137 Oct 30 21:47:22.062406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.062338 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-790a42" Oct 30 21:47:22.062406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.062378 4526 scope.go:122] "RemoveContainer" containerID="9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef" Oct 30 21:47:22.062542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.062523 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7289-9c3aef" podUID="114730a4-766c-4a27-ba62-ae142a65d355" containerName="test-curl-7289-9c3aef" containerID="cri-o://c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58" gracePeriod=2 Oct 30 21:47:22.073601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.073580 4526 scope.go:122] "RemoveContainer" containerID="9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef" Oct 30 21:47:22.073909 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:22.073877 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef\": container with ID starting with 9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef not found: ID does not exist" containerID="9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef" Oct 30 21:47:22.073985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.073906 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef"} err="failed to get container status \"9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef\": rpc error: code = NotFound desc = could not find container \"9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef\": container with ID starting with 9f255167df4fc0994ea1dea62bff1805f98995e3f9cc6c0a288b1934a58f59ef not found: ID does not exist" Oct 30 21:47:22.268913 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.268886 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb"] Oct 30 21:47:22.269720 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:22.269693 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04cdcfe_02d9_4dc8_86da_2774f70f24d8.slice/crio-7dbbf56d70a9524fa4374bbe302ce27fc89c8c173906abea8d65c2b800b90f3d WatchSource:0}: Error finding container 7dbbf56d70a9524fa4374bbe302ce27fc89c8c173906abea8d65c2b800b90f3d: Status 404 returned error can't find the container with id 7dbbf56d70a9524fa4374bbe302ce27fc89c8c173906abea8d65c2b800b90f3d Oct 30 21:47:22.462230 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:22.462204 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7289-3a6fb6"] Oct 30 21:47:22.463702 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:22.463673 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52429b8f_fff8_4b23_a5e5_ae644cd8333e.slice/crio-69afcee393f6d1ccd71c3c6d80623a112e51a1dd7c14a43791f5b56853cc712d WatchSource:0}: Error finding container 69afcee393f6d1ccd71c3c6d80623a112e51a1dd7c14a43791f5b56853cc712d: Status 404 returned error can't find the container with id 69afcee393f6d1ccd71c3c6d80623a112e51a1dd7c14a43791f5b56853cc712d Oct 30 21:47:23.070952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:23.070898 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" event={"ID":"b04cdcfe-02d9-4dc8-86da-2774f70f24d8","Type":"ContainerStarted","Data":"e93b17493fee417b4f6528178c4e53a43cde41cd294d1f86fa24e02d6735959a"} Oct 30 21:47:23.071134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:23.070964 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" event={"ID":"b04cdcfe-02d9-4dc8-86da-2774f70f24d8","Type":"ContainerStarted","Data":"7dbbf56d70a9524fa4374bbe302ce27fc89c8c173906abea8d65c2b800b90f3d"} Oct 30 21:47:23.072562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:23.072535 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-3a6fb6" event={"ID":"52429b8f-fff8-4b23-a5e5-ae644cd8333e","Type":"ContainerStarted","Data":"6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd"} Oct 30 21:47:23.072562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:23.072571 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7289-3a6fb6" event={"ID":"52429b8f-fff8-4b23-a5e5-ae644cd8333e","Type":"ContainerStarted","Data":"69afcee393f6d1ccd71c3c6d80623a112e51a1dd7c14a43791f5b56853cc712d"} Oct 30 21:47:23.187876 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:23.187812 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" podStartSLOduration=2.1877982400000002 podStartE2EDuration="2.18779824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:23.184122212 +0000 UTC m=+1833.317495412" watchObservedRunningTime="2026-10-30 21:47:23.18779824 +0000 UTC m=+1833.321171438" Oct 30 21:47:23.298880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:23.298827 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7289-3a6fb6" podStartSLOduration=2.298812432 podStartE2EDuration="2.298812432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:23.29681046 +0000 UTC m=+1833.430183660" watchObservedRunningTime="2026-10-30 21:47:23.298812432 +0000 UTC m=+1833.432185655" Oct 30 21:47:24.085828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:24.085797 4526 generic.go:356] "Generic (PLEG): container finished" podID="b04cdcfe-02d9-4dc8-86da-2774f70f24d8" containerID="e93b17493fee417b4f6528178c4e53a43cde41cd294d1f86fa24e02d6735959a" exitCode=0 Oct 30 21:47:24.086237 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:24.085893 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" event={"ID":"b04cdcfe-02d9-4dc8-86da-2774f70f24d8","Type":"ContainerDied","Data":"e93b17493fee417b4f6528178c4e53a43cde41cd294d1f86fa24e02d6735959a"} Oct 30 21:47:24.378913 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:24.378888 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:24.426272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:24.426239 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114730a4-766c-4a27-ba62-ae142a65d355-kube-api-access-q2vmq\" (UniqueName: \"kubernetes.io/projected/114730a4-766c-4a27-ba62-ae142a65d355-kube-api-access-q2vmq\") pod \"114730a4-766c-4a27-ba62-ae142a65d355\" (UID: \"114730a4-766c-4a27-ba62-ae142a65d355\") " Oct 30 21:47:24.428667 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:24.428631 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114730a4-766c-4a27-ba62-ae142a65d355-kube-api-access-q2vmq" pod "114730a4-766c-4a27-ba62-ae142a65d355" (UID: "114730a4-766c-4a27-ba62-ae142a65d355"). InnerVolumeSpecName "kube-api-access-q2vmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:24.527829 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:24.527798 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2vmq\" (UniqueName: \"kubernetes.io/projected/114730a4-766c-4a27-ba62-ae142a65d355-kube-api-access-q2vmq\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:24.783811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:24.783724 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7289-3a6fb6"] Oct 30 21:47:24.788442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:24.788412 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7289-3a6fb6"] Oct 30 21:47:25.094293 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.094258 4526 generic.go:356] "Generic (PLEG): container finished" podID="114730a4-766c-4a27-ba62-ae142a65d355" containerID="c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58" exitCode=137 Oct 30 21:47:25.094814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.094308 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:25.094814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.094359 4526 scope.go:122] "RemoveContainer" containerID="c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58" Oct 30 21:47:25.094814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.094745 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7289-3a6fb6" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" containerName="test-curl-7289-3a6fb6" containerID="cri-o://6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd" gracePeriod=2 Oct 30 21:47:25.105845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.105817 4526 scope.go:122] "RemoveContainer" containerID="c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58" Oct 30 21:47:25.106191 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:25.106167 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58\": container with ID starting with c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58 not found: ID does not exist" containerID="c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58" Oct 30 21:47:25.106270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.106206 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58"} err="failed to get container status \"c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58\": rpc error: code = NotFound desc = could not find container \"c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58\": container with ID starting with c324c36f41c127d7cd2e9c271f2fc7532a59f4261866e613a726ba327deeab58 not found: ID does not exist" Oct 30 21:47:25.145476 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:25.145439 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-3a6fb6\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:25.196416 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:25.196375 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-9c3aef\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="114730a4-766c-4a27-ba62-ae142a65d355" pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:25.247462 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:25.247424 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-3a6fb6\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:25.267803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.267779 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:25.340251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.340222 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-kube-api-access-k8kzx\" (UniqueName: \"kubernetes.io/projected/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-kube-api-access-k8kzx\") pod \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " Oct 30 21:47:25.340434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.340348 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-xks-ca\") pod \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " Oct 30 21:47:25.340533 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.340472 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-openshift-ca\") pod \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\" (UID: \"b04cdcfe-02d9-4dc8-86da-2774f70f24d8\") " Oct 30 21:47:25.340772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.340749 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-openshift-ca" pod "b04cdcfe-02d9-4dc8-86da-2774f70f24d8" (UID: "b04cdcfe-02d9-4dc8-86da-2774f70f24d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:25.341076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.341059 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:25.342643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.342621 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-kube-api-access-k8kzx" pod "b04cdcfe-02d9-4dc8-86da-2774f70f24d8" (UID: "b04cdcfe-02d9-4dc8-86da-2774f70f24d8"). InnerVolumeSpecName "kube-api-access-k8kzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:25.342720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.342626 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-xks-ca" pod "b04cdcfe-02d9-4dc8-86da-2774f70f24d8" (UID: "b04cdcfe-02d9-4dc8-86da-2774f70f24d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:25.435949 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:25.435913 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-9c3aef\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="114730a4-766c-4a27-ba62-ae142a65d355" pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:25.442914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.442882 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:25.442914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.442915 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kzx\" (UniqueName: \"kubernetes.io/projected/b04cdcfe-02d9-4dc8-86da-2774f70f24d8-kube-api-access-k8kzx\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:25.486535 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:25.486480 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-3a6fb6\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:25.888593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.888554 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p"] Oct 30 21:47:25.889280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.889265 4526 cpu_manager.go:414] "containerMap: removing container" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" containerName="test-curl-7289-3a6fb6" Oct 30 21:47:25.889331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.889281 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" containerName="test-curl-7289-3a6fb6" Oct 30 21:47:25.889331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.889291 4526 cpu_manager.go:414] "containerMap: removing container" podUID="b04cdcfe-02d9-4dc8-86da-2774f70f24d8" containerName="revoke-keys" Oct 30 21:47:25.889331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.889296 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04cdcfe-02d9-4dc8-86da-2774f70f24d8" containerName="revoke-keys" Oct 30 21:47:25.889433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.889394 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" containerName="test-curl-7289-3a6fb6" Oct 30 21:47:25.889433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.889402 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04cdcfe-02d9-4dc8-86da-2774f70f24d8" containerName="revoke-keys" Oct 30 21:47:25.893109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.893094 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.952695 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.952652 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-credential-socket\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.953064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.953037 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-envoy\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.953254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.953237 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121879d-c1e6-4baa-91cb-c6c492fd2317-istiod-ca-cert\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.953387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.953372 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-token\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.953566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.953547 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-data\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.953715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.953699 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthhg\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-kube-api-access-zthhg\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.953876 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.953862 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-podinfo\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.954090 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:25.954052 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-9c3aef\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="114730a4-766c-4a27-ba62-ae142a65d355" pod="opendatahub/test-curl-7289-9c3aef" Oct 30 21:47:25.954255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.954238 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b34958-openshift-default-dockercfg-hwsb2\"" Oct 30 21:47:25.954586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.954376 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-socket\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:25.961553 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:25.959586 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-certs\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.000246 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.000209 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114730a4-766c-4a27-ba62-ae142a65d355" path="/var/lib/kubelet/pods/114730a4-766c-4a27-ba62-ae142a65d355/volumes" Oct 30 21:47:26.008011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.007950 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p"] Oct 30 21:47:26.061069 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061039 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-podinfo\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061078 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-socket\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061107 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-certs\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061134 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-credential-socket\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061238 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-envoy\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061318 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121879d-c1e6-4baa-91cb-c6c492fd2317-istiod-ca-cert\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061354 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-token\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061460 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-data\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061531 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthhg\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-kube-api-access-zthhg\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061552 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-certs\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061827 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061460 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-socket\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061827 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061732 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-credential-socket\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.061910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.061879 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-data\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.062107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.062082 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121879d-c1e6-4baa-91cb-c6c492fd2317-istiod-ca-cert\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.064041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.064018 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-envoy\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.064166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.064082 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-podinfo\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.105911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.105865 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" event={"ID":"b04cdcfe-02d9-4dc8-86da-2774f70f24d8","Type":"ContainerDied","Data":"7dbbf56d70a9524fa4374bbe302ce27fc89c8c173906abea8d65c2b800b90f3d"} Oct 30 21:47:26.106368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.105923 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dbbf56d70a9524fa4374bbe302ce27fc89c8c173906abea8d65c2b800b90f3d" Oct 30 21:47:26.106368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.105920 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb" Oct 30 21:47:26.120442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.120406 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-token\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.121153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.121113 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthhg\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-kube-api-access-zthhg\") pod \"e2e-api-a-b34958-openshift-default-559d74f555-htq4p\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.121577 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:26.121553 4526 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7289-3a6fb6\" is forbidden: User \"system:node:ip-10-0-128-26.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-26.us-west-1.compute.internal' and this object" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:26.257329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.254658 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:26.714150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.714118 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p"] Oct 30 21:47:26.715127 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:26.715100 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0121879d_c1e6_4baa_91cb_c6c492fd2317.slice/crio-dee9f295e5b801f1db5d12bde2824c481a6849a578e55cc3a2406400fef920b4 WatchSource:0}: Error finding container dee9f295e5b801f1db5d12bde2824c481a6849a578e55cc3a2406400fef920b4: Status 404 returned error can't find the container with id dee9f295e5b801f1db5d12bde2824c481a6849a578e55cc3a2406400fef920b4 Oct 30 21:47:26.718170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.718141 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:26.718257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.718203 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:26.718257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:26.718231 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:27.116028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.115991 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" event={"ID":"0121879d-c1e6-4baa-91cb-c6c492fd2317","Type":"ContainerStarted","Data":"0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61"} Oct 30 21:47:27.116408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.116057 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" event={"ID":"0121879d-c1e6-4baa-91cb-c6c492fd2317","Type":"ContainerStarted","Data":"dee9f295e5b801f1db5d12bde2824c481a6849a578e55cc3a2406400fef920b4"} Oct 30 21:47:27.251283 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.251222 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" podStartSLOduration=2.251204937 podStartE2EDuration="2.251204937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:27.249190382 +0000 UTC m=+1837.382563592" watchObservedRunningTime="2026-10-30 21:47:27.251204937 +0000 UTC m=+1837.384578137" Oct 30 21:47:27.255658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.255628 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:27.262645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.262622 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:27.373323 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.373291 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:27.492045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.492013 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52429b8f-fff8-4b23-a5e5-ae644cd8333e-kube-api-access-7jg9s\" (UniqueName: \"kubernetes.io/projected/52429b8f-fff8-4b23-a5e5-ae644cd8333e-kube-api-access-7jg9s\") pod \"52429b8f-fff8-4b23-a5e5-ae644cd8333e\" (UID: \"52429b8f-fff8-4b23-a5e5-ae644cd8333e\") " Oct 30 21:47:27.494394 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.494363 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52429b8f-fff8-4b23-a5e5-ae644cd8333e-kube-api-access-7jg9s" pod "52429b8f-fff8-4b23-a5e5-ae644cd8333e" (UID: "52429b8f-fff8-4b23-a5e5-ae644cd8333e"). InnerVolumeSpecName "kube-api-access-7jg9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:27.594479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.594440 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jg9s\" (UniqueName: \"kubernetes.io/projected/52429b8f-fff8-4b23-a5e5-ae644cd8333e-kube-api-access-7jg9s\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:27.988001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:27.987965 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" path="/var/lib/kubelet/pods/52429b8f-fff8-4b23-a5e5-ae644cd8333e/volumes" Oct 30 21:47:28.124952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:28.124909 4526 generic.go:356] "Generic (PLEG): container finished" podID="52429b8f-fff8-4b23-a5e5-ae644cd8333e" containerID="6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd" exitCode=137 Oct 30 21:47:28.124952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:28.124955 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7289-3a6fb6" Oct 30 21:47:28.125481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:28.124988 4526 scope.go:122] "RemoveContainer" containerID="6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd" Oct 30 21:47:28.125481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:28.125364 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:28.126435 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:28.126415 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:47:28.135390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:28.135373 4526 scope.go:122] "RemoveContainer" containerID="6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd" Oct 30 21:47:28.135740 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:28.135719 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd\": container with ID starting with 6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd not found: ID does not exist" containerID="6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd" Oct 30 21:47:28.135803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:28.135750 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd"} err="failed to get container status \"6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd\": rpc error: code = NotFound desc = could not find container \"6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd\": container with ID starting with 6549e5a039fe701f8e5541d25e717189e5b784aa403573bbc441154a6d6223dd not found: ID does not exist" Oct 30 21:47:34.757206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.757122 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp"] Oct 30 21:47:34.762994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.762972 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:34.826512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.825844 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7106f4b4-33a0-40bb-b75a-614d8002c293-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp\" (UID: \"7106f4b4-33a0-40bb-b75a-614d8002c293\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:34.826512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.825903 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9r6d\" (UniqueName: \"kubernetes.io/projected/7106f4b4-33a0-40bb-b75a-614d8002c293-kube-api-access-c9r6d\") pod \"payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp\" (UID: \"7106f4b4-33a0-40bb-b75a-614d8002c293\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:34.849513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.844443 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp"] Oct 30 21:47:34.906756 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.906725 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b34958\"" Oct 30 21:47:34.935617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.911181 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b34958-dockercfg-xtc7k\"" Oct 30 21:47:34.935829 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.932350 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7106f4b4-33a0-40bb-b75a-614d8002c293-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp\" (UID: \"7106f4b4-33a0-40bb-b75a-614d8002c293\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:34.935941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.935869 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9r6d\" (UniqueName: \"kubernetes.io/projected/7106f4b4-33a0-40bb-b75a-614d8002c293-kube-api-access-c9r6d\") pod \"payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp\" (UID: \"7106f4b4-33a0-40bb-b75a-614d8002c293\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:34.936365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.933173 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7106f4b4-33a0-40bb-b75a-614d8002c293-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp\" (UID: \"7106f4b4-33a0-40bb-b75a-614d8002c293\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:34.944512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.943584 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk"] Oct 30 21:47:34.950299 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.948984 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:34.955820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:34.955786 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk"] Oct 30 21:47:35.024567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.009507 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9r6d\" (UniqueName: \"kubernetes.io/projected/7106f4b4-33a0-40bb-b75a-614d8002c293-kube-api-access-c9r6d\") pod \"payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp\" (UID: \"7106f4b4-33a0-40bb-b75a-614d8002c293\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:35.043132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.042694 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83150ced-9ed7-40b2-8472-c9585e6258c8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34958-9786989c7-lcprk\" (UID: \"83150ced-9ed7-40b2-8472-c9585e6258c8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:35.043132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.042886 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xzv\" (UniqueName: \"kubernetes.io/projected/83150ced-9ed7-40b2-8472-c9585e6258c8-kube-api-access-g8xzv\") pod \"payload-processing-e2e-api-a-b34958-9786989c7-lcprk\" (UID: \"83150ced-9ed7-40b2-8472-c9585e6258c8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:35.078328 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.078295 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:35.144799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.144755 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83150ced-9ed7-40b2-8472-c9585e6258c8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34958-9786989c7-lcprk\" (UID: \"83150ced-9ed7-40b2-8472-c9585e6258c8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:35.144983 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.144963 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xzv\" (UniqueName: \"kubernetes.io/projected/83150ced-9ed7-40b2-8472-c9585e6258c8-kube-api-access-g8xzv\") pod \"payload-processing-e2e-api-a-b34958-9786989c7-lcprk\" (UID: \"83150ced-9ed7-40b2-8472-c9585e6258c8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:35.145443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.145423 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83150ced-9ed7-40b2-8472-c9585e6258c8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34958-9786989c7-lcprk\" (UID: \"83150ced-9ed7-40b2-8472-c9585e6258c8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:35.226935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.226866 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xzv\" (UniqueName: \"kubernetes.io/projected/83150ced-9ed7-40b2-8472-c9585e6258c8-kube-api-access-g8xzv\") pod \"payload-processing-e2e-api-a-b34958-9786989c7-lcprk\" (UID: \"83150ced-9ed7-40b2-8472-c9585e6258c8\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:35.285862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.285828 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:35.490121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.490094 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp"] Oct 30 21:47:35.491439 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:35.491414 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7106f4b4_33a0_40bb_b75a_614d8002c293.slice/crio-fd17f6a0f44c53d67be8984f4305b7b32eeea248e53fbabd0cab6770c88cf4c8 WatchSource:0}: Error finding container fd17f6a0f44c53d67be8984f4305b7b32eeea248e53fbabd0cab6770c88cf4c8: Status 404 returned error can't find the container with id fd17f6a0f44c53d67be8984f4305b7b32eeea248e53fbabd0cab6770c88cf4c8 Oct 30 21:47:35.723626 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:35.723594 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83150ced_9ed7_40b2_8472_c9585e6258c8.slice/crio-b343f81160d1a2b8bbf539c411e39601121cfaae824c2d6ccd873f2be0fd8627 WatchSource:0}: Error finding container b343f81160d1a2b8bbf539c411e39601121cfaae824c2d6ccd873f2be0fd8627: Status 404 returned error can't find the container with id b343f81160d1a2b8bbf539c411e39601121cfaae824c2d6ccd873f2be0fd8627 Oct 30 21:47:35.724990 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:35.724964 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk"] Oct 30 21:47:36.191211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:36.191164 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" event={"ID":"7106f4b4-33a0-40bb-b75a-614d8002c293","Type":"ContainerStarted","Data":"0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55"} Oct 30 21:47:36.192066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:36.191236 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" event={"ID":"7106f4b4-33a0-40bb-b75a-614d8002c293","Type":"ContainerStarted","Data":"fd17f6a0f44c53d67be8984f4305b7b32eeea248e53fbabd0cab6770c88cf4c8"} Oct 30 21:47:36.192066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:36.191262 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:36.193241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:36.193213 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" event={"ID":"83150ced-9ed7-40b2-8472-c9585e6258c8","Type":"ContainerStarted","Data":"3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908"} Oct 30 21:47:36.193380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:36.193260 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" event={"ID":"83150ced-9ed7-40b2-8472-c9585e6258c8","Type":"ContainerStarted","Data":"b343f81160d1a2b8bbf539c411e39601121cfaae824c2d6ccd873f2be0fd8627"} Oct 30 21:47:36.193380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:36.193286 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:36.319302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:36.319241 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" podStartSLOduration=2.319222494 podStartE2EDuration="2.319222494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:36.317916124 +0000 UTC m=+1846.451289323" watchObservedRunningTime="2026-10-30 21:47:36.319222494 +0000 UTC m=+1846.452595693" Oct 30 21:47:36.434873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:36.434816 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" podStartSLOduration=2.4347961639999998 podStartE2EDuration="2.434796164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:36.431376793 +0000 UTC m=+1846.564750014" watchObservedRunningTime="2026-10-30 21:47:36.434796164 +0000 UTC m=+1846.568169380" Oct 30 21:47:40.820197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.820158 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx"] Oct 30 21:47:40.825588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.825560 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:40.877399 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.877365 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:40.927134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.927092 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:47:40.927134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.927092 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:47:40.927393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.927234 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:47:40.951644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.951608 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx"] Oct 30 21:47:40.953778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.953753 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:40.953933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.953820 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:40.953980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:40.953944 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576gm\" (UniqueName: \"kubernetes.io/projected/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-kube-api-access-576gm\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:41.055722 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:41.055688 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:41.055941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:41.055776 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:41.055941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:41.055850 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576gm\" (UniqueName: \"kubernetes.io/projected/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-kube-api-access-576gm\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:41.056561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:41.056533 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:41.058963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:41.058939 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:41.113561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:41.113448 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576gm\" (UniqueName: \"kubernetes.io/projected/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-kube-api-access-576gm\") pod \"maas-api-revoke-sub-e8c5afda-ttmjx\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:41.143919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:41.143888 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:41.536332 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:41.536301 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx"] Oct 30 21:47:41.537026 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:47:41.537000 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc5889f_5d8f_4050_aecb_ac95b21e9ce2.slice/crio-90b1c109f738afe0016003145bfbf9bdeed356f237580b9871f17dfa13d867cd WatchSource:0}: Error finding container 90b1c109f738afe0016003145bfbf9bdeed356f237580b9871f17dfa13d867cd: Status 404 returned error can't find the container with id 90b1c109f738afe0016003145bfbf9bdeed356f237580b9871f17dfa13d867cd Oct 30 21:47:42.265382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:42.265298 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" event={"ID":"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2","Type":"ContainerStarted","Data":"c116c9bbaa7d93228c1dc0af9983fbef6e9684b4065deea5bd95cc060ff327b6"} Oct 30 21:47:42.265382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:42.265355 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" event={"ID":"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2","Type":"ContainerStarted","Data":"90b1c109f738afe0016003145bfbf9bdeed356f237580b9871f17dfa13d867cd"} Oct 30 21:47:42.382090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:42.382028 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" podStartSLOduration=2.38201004 podStartE2EDuration="2.38201004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:42.381702176 +0000 UTC m=+1852.515075373" watchObservedRunningTime="2026-10-30 21:47:42.38201004 +0000 UTC m=+1852.515383239" Oct 30 21:47:42.480766 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:47:42.480740 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:47:43.127209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.125281 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:47:43.127209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.125333 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:47:43.274797 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.274723 4526 generic.go:356] "Generic (PLEG): container finished" podID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerID="0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58" exitCode=0 Oct 30 21:47:43.275259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.274792 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" event={"ID":"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb","Type":"ContainerDied","Data":"0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58"} Oct 30 21:47:43.275346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.275287 4526 scope.go:122] "RemoveContainer" containerID="0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58" Oct 30 21:47:43.276625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.276600 4526 generic.go:356] "Generic (PLEG): container finished" podID="7fc5889f-5d8f-4050-aecb-ac95b21e9ce2" containerID="c116c9bbaa7d93228c1dc0af9983fbef6e9684b4065deea5bd95cc060ff327b6" exitCode=0 Oct 30 21:47:43.276807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.276791 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" event={"ID":"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2","Type":"ContainerDied","Data":"c116c9bbaa7d93228c1dc0af9983fbef6e9684b4065deea5bd95cc060ff327b6"} Oct 30 21:47:43.277182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.277151 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:43.277257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.277241 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:43.277322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:43.277290 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:44.286446 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.286409 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" event={"ID":"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb","Type":"ContainerStarted","Data":"8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68"} Oct 30 21:47:44.525590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.525559 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:44.628013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.627312 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-kube-api-access-576gm\" (UniqueName: \"kubernetes.io/projected/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-kube-api-access-576gm\") pod \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " Oct 30 21:47:44.628013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.627636 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-openshift-ca\") pod \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " Oct 30 21:47:44.628013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.627786 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-xks-ca\") pod \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\" (UID: \"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2\") " Oct 30 21:47:44.630211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.629068 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-openshift-ca" pod "7fc5889f-5d8f-4050-aecb-ac95b21e9ce2" (UID: "7fc5889f-5d8f-4050-aecb-ac95b21e9ce2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:44.632043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.632012 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-xks-ca" pod "7fc5889f-5d8f-4050-aecb-ac95b21e9ce2" (UID: "7fc5889f-5d8f-4050-aecb-ac95b21e9ce2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:44.633243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.633220 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-kube-api-access-576gm" pod "7fc5889f-5d8f-4050-aecb-ac95b21e9ce2" (UID: "7fc5889f-5d8f-4050-aecb-ac95b21e9ce2"). InnerVolumeSpecName "kube-api-access-576gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:44.730078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.730039 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:44.730078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.730071 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576gm\" (UniqueName: \"kubernetes.io/projected/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-kube-api-access-576gm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:44.730078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:44.730085 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.203077 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:45.202988 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:47:45.203405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:45.203381 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:45.203607 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:45.203584 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:45.319767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:45.319732 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" Oct 30 21:47:45.319767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:45.319755 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx" event={"ID":"7fc5889f-5d8f-4050-aecb-ac95b21e9ce2","Type":"ContainerDied","Data":"90b1c109f738afe0016003145bfbf9bdeed356f237580b9871f17dfa13d867cd"} Oct 30 21:47:45.320224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:45.319808 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b1c109f738afe0016003145bfbf9bdeed356f237580b9871f17dfa13d867cd" Oct 30 21:47:46.214514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:46.203843 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:46.214514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:46.203902 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:47.216839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:47.216805 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:47.217442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:47.217408 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:47.228010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:47.227985 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:47:47.228146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:47.228061 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:47:48.208514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:48.200842 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:48.208514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:48.200914 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:49.205514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:49.197918 4526 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:47:49.205514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:49.197994 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 21:47:50.202260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:50.202225 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:47:50.202615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:50.202593 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:47:50.203343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:50.203321 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:47:57.935899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:57.935854 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz"] Oct 30 21:47:57.936629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:57.936611 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7fc5889f-5d8f-4050-aecb-ac95b21e9ce2" containerName="revoke-keys" Oct 30 21:47:57.936629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:57.936628 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc5889f-5d8f-4050-aecb-ac95b21e9ce2" containerName="revoke-keys" Oct 30 21:47:57.936801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:57.936768 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc5889f-5d8f-4050-aecb-ac95b21e9ce2" containerName="revoke-keys" Oct 30 21:47:57.942273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:57.942250 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.000179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.000143 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz"] Oct 30 21:47:58.026586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.026531 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ghf\" (UniqueName: \"kubernetes.io/projected/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-kube-api-access-29ghf\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.026811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.026603 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.026888 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.026810 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.051953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.051780 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:58.051953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.051922 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:47:58.052627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.052589 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:47:58.052942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.052917 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:47:58.128566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.128461 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.128765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.128733 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ghf\" (UniqueName: \"kubernetes.io/projected/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-kube-api-access-29ghf\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.128837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.128775 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.129718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.129692 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.132166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.132137 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.192521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.191432 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ghf\" (UniqueName: \"kubernetes.io/projected/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-kube-api-access-29ghf\") pod \"maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.259054 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.259016 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:47:58.671715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:58.671675 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz"] Oct 30 21:47:59.491516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:59.490459 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" event={"ID":"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802","Type":"ContainerStarted","Data":"9c152563ac8522e5147f39da5e487767f32515e1ec518570e6db2c6163a2923e"} Oct 30 21:47:59.491516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:47:59.490591 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" event={"ID":"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802","Type":"ContainerStarted","Data":"0332d81cdf49bace3b22efd49056aa366e30817965c991d8ce03e6445a5a2975"} Oct 30 21:48:00.502703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:00.502666 4526 generic.go:356] "Generic (PLEG): container finished" podID="7f0d77a0-35f4-4db3-b2bf-0023a0dbf802" containerID="9c152563ac8522e5147f39da5e487767f32515e1ec518570e6db2c6163a2923e" exitCode=0 Oct 30 21:48:00.503232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:00.502773 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" event={"ID":"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802","Type":"ContainerDied","Data":"9c152563ac8522e5147f39da5e487767f32515e1ec518570e6db2c6163a2923e"} Oct 30 21:48:01.678653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.678629 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:48:01.842272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.842229 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-xks-ca\") pod \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " Oct 30 21:48:01.842482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.842328 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-openshift-ca\") pod \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " Oct 30 21:48:01.842482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.842442 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-kube-api-access-29ghf\" (UniqueName: \"kubernetes.io/projected/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-kube-api-access-29ghf\") pod \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\" (UID: \"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802\") " Oct 30 21:48:01.843930 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.843897 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-openshift-ca" pod "7f0d77a0-35f4-4db3-b2bf-0023a0dbf802" (UID: "7f0d77a0-35f4-4db3-b2bf-0023a0dbf802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:01.846275 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.846243 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-kube-api-access-29ghf" pod "7f0d77a0-35f4-4db3-b2bf-0023a0dbf802" (UID: "7f0d77a0-35f4-4db3-b2bf-0023a0dbf802"). InnerVolumeSpecName "kube-api-access-29ghf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:01.846623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.846603 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-xks-ca" pod "7f0d77a0-35f4-4db3-b2bf-0023a0dbf802" (UID: "7f0d77a0-35f4-4db3-b2bf-0023a0dbf802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:01.944556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.944518 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ghf\" (UniqueName: \"kubernetes.io/projected/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-kube-api-access-29ghf\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:01.944556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.944550 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:01.944556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:01.944560 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:02.523560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:02.523532 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" Oct 30 21:48:02.523711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:02.523539 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz" event={"ID":"7f0d77a0-35f4-4db3-b2bf-0023a0dbf802","Type":"ContainerDied","Data":"0332d81cdf49bace3b22efd49056aa366e30817965c991d8ce03e6445a5a2975"} Oct 30 21:48:02.523711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:02.523656 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0332d81cdf49bace3b22efd49056aa366e30817965c991d8ce03e6445a5a2975" Oct 30 21:48:09.423258 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:09.423226 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8"] Oct 30 21:48:09.537886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:09.537853 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz"] Oct 30 21:48:12.479364 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:48:12.479327 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:48:14.424529 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.424464 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" podUID="22761589-b83d-4520-901c-3d8eef2b3421" containerName="payload-processing" containerID="cri-o://c6cddec16bfb3b142d36eb9574664e8fc3081c2fd05d726ebb2068088ec2740f" gracePeriod=25 Oct 30 21:48:14.539202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.539161 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" podUID="09e1c745-932a-423e-9abe-3b4f5b1d7246" containerName="payload-pre-processing" containerID="cri-o://cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee" gracePeriod=25 Oct 30 21:48:14.616655 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.616613 4526 generic.go:356] "Generic (PLEG): container finished" podID="22761589-b83d-4520-901c-3d8eef2b3421" containerID="c6cddec16bfb3b142d36eb9574664e8fc3081c2fd05d726ebb2068088ec2740f" exitCode=0 Oct 30 21:48:14.616865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.616845 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" event={"ID":"22761589-b83d-4520-901c-3d8eef2b3421","Type":"ContainerDied","Data":"c6cddec16bfb3b142d36eb9574664e8fc3081c2fd05d726ebb2068088ec2740f"} Oct 30 21:48:14.727985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.727961 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:48:14.837968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.837942 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:48:14.856980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.856952 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22761589-b83d-4520-901c-3d8eef2b3421-kube-api-access-lbzhb\" (UniqueName: \"kubernetes.io/projected/22761589-b83d-4520-901c-3d8eef2b3421-kube-api-access-lbzhb\") pod \"22761589-b83d-4520-901c-3d8eef2b3421\" (UID: \"22761589-b83d-4520-901c-3d8eef2b3421\") " Oct 30 21:48:14.857120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.857067 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22761589-b83d-4520-901c-3d8eef2b3421-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22761589-b83d-4520-901c-3d8eef2b3421-plugins-config-volume\") pod \"22761589-b83d-4520-901c-3d8eef2b3421\" (UID: \"22761589-b83d-4520-901c-3d8eef2b3421\") " Oct 30 21:48:14.857434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.857411 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22761589-b83d-4520-901c-3d8eef2b3421-plugins-config-volume" pod "22761589-b83d-4520-901c-3d8eef2b3421" (UID: "22761589-b83d-4520-901c-3d8eef2b3421"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:14.857624 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.857604 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22761589-b83d-4520-901c-3d8eef2b3421-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.859378 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.859350 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22761589-b83d-4520-901c-3d8eef2b3421-kube-api-access-lbzhb" pod "22761589-b83d-4520-901c-3d8eef2b3421" (UID: "22761589-b83d-4520-901c-3d8eef2b3421"). InnerVolumeSpecName "kube-api-access-lbzhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:14.958477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.958385 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e1c745-932a-423e-9abe-3b4f5b1d7246-kube-api-access-spmq9\" (UniqueName: \"kubernetes.io/projected/09e1c745-932a-423e-9abe-3b4f5b1d7246-kube-api-access-spmq9\") pod \"09e1c745-932a-423e-9abe-3b4f5b1d7246\" (UID: \"09e1c745-932a-423e-9abe-3b4f5b1d7246\") " Oct 30 21:48:14.958477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.958468 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e1c745-932a-423e-9abe-3b4f5b1d7246-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e1c745-932a-423e-9abe-3b4f5b1d7246-plugins-config-volume\") pod \"09e1c745-932a-423e-9abe-3b4f5b1d7246\" (UID: \"09e1c745-932a-423e-9abe-3b4f5b1d7246\") " Oct 30 21:48:14.959172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.958890 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e1c745-932a-423e-9abe-3b4f5b1d7246-plugins-config-volume" pod "09e1c745-932a-423e-9abe-3b4f5b1d7246" (UID: "09e1c745-932a-423e-9abe-3b4f5b1d7246"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:14.959313 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.959291 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e1c745-932a-423e-9abe-3b4f5b1d7246-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.959405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.959320 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbzhb\" (UniqueName: \"kubernetes.io/projected/22761589-b83d-4520-901c-3d8eef2b3421-kube-api-access-lbzhb\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:14.962038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:14.962008 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e1c745-932a-423e-9abe-3b4f5b1d7246-kube-api-access-spmq9" pod "09e1c745-932a-423e-9abe-3b4f5b1d7246" (UID: "09e1c745-932a-423e-9abe-3b4f5b1d7246"). InnerVolumeSpecName "kube-api-access-spmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:15.060395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.060354 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spmq9\" (UniqueName: \"kubernetes.io/projected/09e1c745-932a-423e-9abe-3b4f5b1d7246-kube-api-access-spmq9\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:15.625316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.625286 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" Oct 30 21:48:15.625316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.625296 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8" event={"ID":"22761589-b83d-4520-901c-3d8eef2b3421","Type":"ContainerDied","Data":"3c01ab3d70c73d22590291b3f01460fc29d254163d742aef5673525cff14e05c"} Oct 30 21:48:15.625927 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.625397 4526 scope.go:122] "RemoveContainer" containerID="c6cddec16bfb3b142d36eb9574664e8fc3081c2fd05d726ebb2068088ec2740f" Oct 30 21:48:15.626841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.626793 4526 generic.go:356] "Generic (PLEG): container finished" podID="09e1c745-932a-423e-9abe-3b4f5b1d7246" containerID="cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee" exitCode=0 Oct 30 21:48:15.626929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.626861 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" Oct 30 21:48:15.626929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.626872 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" event={"ID":"09e1c745-932a-423e-9abe-3b4f5b1d7246","Type":"ContainerDied","Data":"cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee"} Oct 30 21:48:15.626929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.626914 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz" event={"ID":"09e1c745-932a-423e-9abe-3b4f5b1d7246","Type":"ContainerDied","Data":"f5c44b7d4438c033d108725c537d917fbc0299d94f4645e2ad7becc95739ab9b"} Oct 30 21:48:15.637130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.637118 4526 scope.go:122] "RemoveContainer" containerID="cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee" Oct 30 21:48:15.646727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.646710 4526 scope.go:122] "RemoveContainer" containerID="cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee" Oct 30 21:48:15.647004 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:48:15.646985 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee\": container with ID starting with cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee not found: ID does not exist" containerID="cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee" Oct 30 21:48:15.647056 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.647014 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee"} err="failed to get container status \"cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee\": rpc error: code = NotFound desc = could not find container \"cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee\": container with ID starting with cf611bf607816d77718b25a4ea3befa22c98461d307fc39253f34692992542ee not found: ID does not exist" Oct 30 21:48:15.790125 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.790083 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8"] Oct 30 21:48:15.838801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.838770 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-29b7ead1-747f6b9c57-5twv8"] Oct 30 21:48:15.946844 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.946805 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz"] Oct 30 21:48:15.950430 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.950406 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-29b7ead1-74ffb48bb7-tt5qz"] Oct 30 21:48:15.987609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.987578 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e1c745-932a-423e-9abe-3b4f5b1d7246" path="/var/lib/kubelet/pods/09e1c745-932a-423e-9abe-3b4f5b1d7246/volumes" Oct 30 21:48:15.988028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:15.988015 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22761589-b83d-4520-901c-3d8eef2b3421" path="/var/lib/kubelet/pods/22761589-b83d-4520-901c-3d8eef2b3421/volumes" Oct 30 21:48:17.365563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.365519 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz"] Oct 30 21:48:17.366445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366422 4526 cpu_manager.go:414] "containerMap: removing container" podUID="09e1c745-932a-423e-9abe-3b4f5b1d7246" containerName="payload-pre-processing" Oct 30 21:48:17.366579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366447 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e1c745-932a-423e-9abe-3b4f5b1d7246" containerName="payload-pre-processing" Oct 30 21:48:17.366579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366478 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7f0d77a0-35f4-4db3-b2bf-0023a0dbf802" containerName="revoke-keys" Oct 30 21:48:17.366579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366505 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0d77a0-35f4-4db3-b2bf-0023a0dbf802" containerName="revoke-keys" Oct 30 21:48:17.366579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366530 4526 cpu_manager.go:414] "containerMap: removing container" podUID="22761589-b83d-4520-901c-3d8eef2b3421" containerName="payload-processing" Oct 30 21:48:17.366579 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366538 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22761589-b83d-4520-901c-3d8eef2b3421" containerName="payload-processing" Oct 30 21:48:17.366836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366744 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e1c745-932a-423e-9abe-3b4f5b1d7246" containerName="payload-pre-processing" Oct 30 21:48:17.366836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366763 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0d77a0-35f4-4db3-b2bf-0023a0dbf802" containerName="revoke-keys" Oct 30 21:48:17.366836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.366776 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="22761589-b83d-4520-901c-3d8eef2b3421" containerName="payload-processing" Oct 30 21:48:17.372018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.371994 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.392910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.392878 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.393098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.393080 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.393154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.393122 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.393228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.393202 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.393287 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.393245 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.393339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.393283 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.393428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.393412 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.393509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.393447 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2k9\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-kube-api-access-rk2k9\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.393569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.393540 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.415288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.415252 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l"] Oct 30 21:48:17.419931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.419910 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.425196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.425168 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-3ddec2-openshift-default-dockercfg-bvfmv\"" Oct 30 21:48:17.473544 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.473504 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz"] Oct 30 21:48:17.494478 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.494446 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.494736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.494673 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jtv\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-kube-api-access-49jtv\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.494798 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.494736 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.494798 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.494788 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.494905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.494828 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.495277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.495251 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.495619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.495597 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.496693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496666 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.496825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496715 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2k9\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-kube-api-access-rk2k9\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.496825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496753 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.496825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496824 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.496980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496855 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.496980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496875 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.496980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496902 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.496980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496957 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.497178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.496999 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.497178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.497022 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.497178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.497064 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.497178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.497091 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.497178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.497125 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.497574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.497552 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.498531 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.498068 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.498698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.497791 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.500824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.500796 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.501755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.501731 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.527973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.527941 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l"] Oct 30 21:48:17.555926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.555892 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.555926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.555916 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2k9\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-kube-api-access-rk2k9\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.598706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.598668 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.598916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.598727 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.598916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.598752 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.598916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.598783 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.598916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.598884 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.598916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.598916 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.599197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.599000 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jtv\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-kube-api-access-49jtv\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.599197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.599063 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.599197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.599097 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.599968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.599503 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.599968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.599525 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.599968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.599683 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.599968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.599874 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.599968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.599931 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.602066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.602012 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.603193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.602899 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.656200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.656129 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.656403 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.656383 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jtv\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-kube-api-access-49jtv\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:17.696211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.696151 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:17.735223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:17.734887 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:18.138944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.138914 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz"] Oct 30 21:48:18.139057 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:48:18.138999 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb6d36a_acfa_410e_b3e3_c8fc1102ec1f.slice/crio-39ee059d681d12163b0334a8f0f804138aae6190954adc41b228c194b4dd4afd WatchSource:0}: Error finding container 39ee059d681d12163b0334a8f0f804138aae6190954adc41b228c194b4dd4afd: Status 404 returned error can't find the container with id 39ee059d681d12163b0334a8f0f804138aae6190954adc41b228c194b4dd4afd Oct 30 21:48:18.145235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.143365 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.145235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.143558 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.145235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.143771 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.145965 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.145933 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l"] Oct 30 21:48:18.190917 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:48:18.190880 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8644e88_b41d_4e2e_ae13_b8d6f2b8bfcf.slice/crio-b1bafdbf12a9f94d0ac476303efe07736f150eaad1c3c9aaa88a7e7f5a0de62a WatchSource:0}: Error finding container b1bafdbf12a9f94d0ac476303efe07736f150eaad1c3c9aaa88a7e7f5a0de62a: Status 404 returned error can't find the container with id b1bafdbf12a9f94d0ac476303efe07736f150eaad1c3c9aaa88a7e7f5a0de62a Oct 30 21:48:18.193790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.193756 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.193873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.193830 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.193873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.193861 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.654640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.654603 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" event={"ID":"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f","Type":"ContainerStarted","Data":"55f5cc51c894fd7e376cde387c8a4879bf143299876fdbea38c640f5b81ceeaf"} Oct 30 21:48:18.655105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.654664 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" event={"ID":"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f","Type":"ContainerStarted","Data":"39ee059d681d12163b0334a8f0f804138aae6190954adc41b228c194b4dd4afd"} Oct 30 21:48:18.656713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.656676 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" event={"ID":"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf","Type":"ContainerStarted","Data":"dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2"} Oct 30 21:48:18.656826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.656731 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" event={"ID":"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf","Type":"ContainerStarted","Data":"b1bafdbf12a9f94d0ac476303efe07736f150eaad1c3c9aaa88a7e7f5a0de62a"} Oct 30 21:48:18.696906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.696870 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:18.735787 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.735760 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:18.776968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.776917 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podStartSLOduration=1.776901729 podStartE2EDuration="1.776901729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:18.773298622 +0000 UTC m=+1888.906671818" watchObservedRunningTime="2026-10-30 21:48:18.776901729 +0000 UTC m=+1888.910274928" Oct 30 21:48:18.890468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:18.890401 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" podStartSLOduration=1.890384786 podStartE2EDuration="1.890384786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:18.889459124 +0000 UTC m=+1889.022832347" watchObservedRunningTime="2026-10-30 21:48:18.890384786 +0000 UTC m=+1889.023757984" Oct 30 21:48:19.720528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:19.719638 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 21:48:19.720528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:19.719707 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 21:48:19.745855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:19.745748 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 21:48:19.745855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:19.745802 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 21:48:20.743988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:20.743946 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 21:48:20.744664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:20.744621 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 21:48:20.748540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:20.745444 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 21:48:20.748540 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:20.747387 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 21:48:21.700525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:21.699934 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 21:48:21.700525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:21.699999 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 21:48:21.739516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:21.738939 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 21:48:21.739516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:21.739002 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 21:48:22.701514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:22.697864 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 21:48:22.701514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:22.697926 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 21:48:22.743096 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:22.742944 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 21:48:22.743096 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:22.743004 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 21:48:23.716512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:23.713084 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 21:48:23.716512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:23.713243 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 21:48:23.743659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:23.743627 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 21:48:23.743933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:23.743903 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 21:48:24.697159 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:24.697112 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 21:48:24.697386 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:24.697174 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 21:48:24.809361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:24.809327 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:24.810598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:24.810572 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:24.812580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:24.812555 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:48:25.701733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:25.701703 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:25.702173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:25.702154 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:25.702804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:25.702787 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:48:27.953191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:27.953158 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm"] Oct 30 21:48:27.957893 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:27.957871 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.015510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.010469 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-013fb2-openshift-default-dockercfg-qvxxw\"" Oct 30 21:48:28.043837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.043794 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-token\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.044268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.044245 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istiod-ca-cert\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.060511 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.048606 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-certs\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.089514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.070941 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-podinfo\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.089879 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.089848 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-credential-socket\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.101625 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.101590 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2748\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-kube-api-access-q2748\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.103935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.066607 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm"] Oct 30 21:48:28.103935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.101929 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-data\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.103935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.102127 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-socket\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.103935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.102190 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-envoy\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.204654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.204527 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-token\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.204654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.204649 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istiod-ca-cert\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.204674 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-certs\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.204729 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-podinfo\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.204789 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-credential-socket\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.204831 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2748\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-kube-api-access-q2748\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.204927 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-data\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.205032 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-socket\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.205095 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-envoy\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.205242 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-certs\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.205648 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-socket\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.206150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.206010 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-credential-socket\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.207016 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.206990 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istiod-ca-cert\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.207824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.207802 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-data\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.208357 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.208332 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-podinfo\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.209953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.209932 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-envoy\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.262171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.262134 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-token\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.262690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.262666 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2748\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-kube-api-access-q2748\") pod \"e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.274634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.274601 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:28.759888 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.759850 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm"] Oct 30 21:48:28.765905 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:48:28.765872 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67bb07b_9ee6_412a_a850_c872cb33b1b2.slice/crio-5e967fb86777611188847d7a2e7de6258bb06b6ecbd3ddc80b3da2649cb9f843 WatchSource:0}: Error finding container 5e967fb86777611188847d7a2e7de6258bb06b6ecbd3ddc80b3da2649cb9f843: Status 404 returned error can't find the container with id 5e967fb86777611188847d7a2e7de6258bb06b6ecbd3ddc80b3da2649cb9f843 Oct 30 21:48:28.770779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.770723 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:28.771182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.771069 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:28.771182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.771152 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:28.801436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:28.801343 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" event={"ID":"c67bb07b-9ee6-412a-a850-c872cb33b1b2","Type":"ContainerStarted","Data":"5e967fb86777611188847d7a2e7de6258bb06b6ecbd3ddc80b3da2649cb9f843"} Oct 30 21:48:29.827514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:29.820867 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" event={"ID":"c67bb07b-9ee6-412a-a850-c872cb33b1b2","Type":"ContainerStarted","Data":"06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544"} Oct 30 21:48:30.278156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:30.278067 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:30.295812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:30.295782 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:30.422347 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:30.422283 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" podStartSLOduration=3.422262271 podStartE2EDuration="3.422262271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:29.95195129 +0000 UTC m=+1900.085324489" watchObservedRunningTime="2026-10-30 21:48:30.422262271 +0000 UTC m=+1900.555635470" Oct 30 21:48:30.952516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:30.951559 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:30.987374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:30.987344 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:48:32.251528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:32.251479 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz"] Oct 30 21:48:32.287516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:32.287451 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-29b7ead1-c5lhz"] Oct 30 21:48:33.986548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:33.986515 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0d77a0-35f4-4db3-b2bf-0023a0dbf802" path="/var/lib/kubelet/pods/7f0d77a0-35f4-4db3-b2bf-0023a0dbf802/volumes" Oct 30 21:48:39.060678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:39.060646 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd"] Oct 30 21:48:39.061068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:39.060901 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" containerID="cri-o://61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279" gracePeriod=30 Oct 30 21:48:42.127816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.127763 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg"] Oct 30 21:48:42.133080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.133057 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:42.195513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.195463 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg"] Oct 30 21:48:42.290513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.271010 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4a1551\"" Oct 30 21:48:42.339161 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.324733 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4a1551-dockercfg-k7r6l\"" Oct 30 21:48:42.339438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.337771 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc"] Oct 30 21:48:42.341754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.341721 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg\" (UID: \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:42.341976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.341961 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8flq\" (UniqueName: \"kubernetes.io/projected/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-kube-api-access-s8flq\") pod \"payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg\" (UID: \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:42.391419 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.391342 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc"] Oct 30 21:48:42.392985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.392961 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:42.444252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.444215 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg\" (UID: \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:42.444438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.444269 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8flq\" (UniqueName: \"kubernetes.io/projected/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-kube-api-access-s8flq\") pod \"payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg\" (UID: \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:42.444481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.444443 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc\" (UID: \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:42.444607 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.444589 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbbv\" (UniqueName: \"kubernetes.io/projected/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-kube-api-access-2dbbv\") pod \"payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc\" (UID: \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:42.445192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.445173 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg\" (UID: \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:42.481470 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:48:42.481426 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:48:42.503652 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.503623 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8flq\" (UniqueName: \"kubernetes.io/projected/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-kube-api-access-s8flq\") pod \"payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg\" (UID: \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:42.545898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.545861 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc\" (UID: \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:42.546077 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.545946 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbbv\" (UniqueName: \"kubernetes.io/projected/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-kube-api-access-2dbbv\") pod \"payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc\" (UID: \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:42.546674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.546654 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc\" (UID: \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:42.603576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.603548 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbbv\" (UniqueName: \"kubernetes.io/projected/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-kube-api-access-2dbbv\") pod \"payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc\" (UID: \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:42.718215 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.718114 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:42.748743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:42.748708 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:43.110147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:43.110119 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc"] Oct 30 21:48:43.113421 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:48:43.113386 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bae8ebc_d123_4e75_9e1a_1b2b343a70d1.slice/crio-e58eda49dc0d063c95af84b1c1d6aa654ca85740328d807f88f6d07487a79911 WatchSource:0}: Error finding container e58eda49dc0d063c95af84b1c1d6aa654ca85740328d807f88f6d07487a79911: Status 404 returned error can't find the container with id e58eda49dc0d063c95af84b1c1d6aa654ca85740328d807f88f6d07487a79911 Oct 30 21:48:43.212029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:43.208903 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg"] Oct 30 21:48:43.215813 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:48:43.215779 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d5325c_bcc7_4a13_b2ac_286cfc8ae6a5.slice/crio-47a82f410d32bd30cc359953b10a267500eeac0ba1df7d04807e5192f4f8ae08 WatchSource:0}: Error finding container 47a82f410d32bd30cc359953b10a267500eeac0ba1df7d04807e5192f4f8ae08: Status 404 returned error can't find the container with id 47a82f410d32bd30cc359953b10a267500eeac0ba1df7d04807e5192f4f8ae08 Oct 30 21:48:44.060796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.060762 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" event={"ID":"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5","Type":"ContainerStarted","Data":"c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05"} Oct 30 21:48:44.061009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.060816 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" event={"ID":"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5","Type":"ContainerStarted","Data":"47a82f410d32bd30cc359953b10a267500eeac0ba1df7d04807e5192f4f8ae08"} Oct 30 21:48:44.061009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.060854 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:44.062651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.062619 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" event={"ID":"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1","Type":"ContainerStarted","Data":"39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16"} Oct 30 21:48:44.062806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.062683 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:44.062806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.062710 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" event={"ID":"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1","Type":"ContainerStarted","Data":"e58eda49dc0d063c95af84b1c1d6aa654ca85740328d807f88f6d07487a79911"} Oct 30 21:48:44.174124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.174076 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" podStartSLOduration=2.174061343 podStartE2EDuration="2.174061343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:44.171654673 +0000 UTC m=+1914.305027871" watchObservedRunningTime="2026-10-30 21:48:44.174061343 +0000 UTC m=+1914.307434540" Oct 30 21:48:44.300532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.299233 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" podStartSLOduration=2.299207494 podStartE2EDuration="2.299207494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:44.282890732 +0000 UTC m=+1914.416263931" watchObservedRunningTime="2026-10-30 21:48:44.299207494 +0000 UTC m=+1914.432580703" Oct 30 21:48:44.388387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.388362 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:48:44.477762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.477728 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-credential-socket\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.477969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.477779 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-data\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.477969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.477856 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-envoy\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.477969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.477963 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-kube-api-access-rgftq\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-kube-api-access-rgftq\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.478144 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.477996 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istiod-ca-cert\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.478144 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478045 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-certs\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.478144 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478046 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-data" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:44.478144 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478067 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-credential-socket" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:44.478144 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478078 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-socket\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.478397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478151 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-token\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.478397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478274 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-certs" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:44.478397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478365 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-socket" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:44.478397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478374 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istiod-ca-cert" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:44.478656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.478384 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-podinfo\") pod \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\" (UID: \"0cc8fdad-213d-4748-94f3-f4c1359c8bbc\") " Oct 30 21:48:44.479103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.479081 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:44.479103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.479103 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:44.479301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.479113 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:44.479301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.479122 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:44.479301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.479134 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:44.480538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.480466 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-kube-api-access-rgftq" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "kube-api-access-rgftq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:44.480538 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.480530 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-podinfo" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:44.480781 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.480761 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-envoy" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:44.480928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.480908 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-token" pod "0cc8fdad-213d-4748-94f3-f4c1359c8bbc" (UID: "0cc8fdad-213d-4748-94f3-f4c1359c8bbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:44.580270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.580235 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:44.580270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.580265 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:44.580270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.580276 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:44.580533 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:44.580286 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgftq\" (UniqueName: \"kubernetes.io/projected/0cc8fdad-213d-4748-94f3-f4c1359c8bbc-kube-api-access-rgftq\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:45.074508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.073642 4526 generic.go:356] "Generic (PLEG): container finished" podID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerID="61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279" exitCode=0 Oct 30 21:48:45.074508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.073736 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" event={"ID":"0cc8fdad-213d-4748-94f3-f4c1359c8bbc","Type":"ContainerDied","Data":"61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279"} Oct 30 21:48:45.074508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.073793 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" event={"ID":"0cc8fdad-213d-4748-94f3-f4c1359c8bbc","Type":"ContainerDied","Data":"cd071a0ca446f83197bac803ca3c3491c46295c8267a00009c076c1563f6c95d"} Oct 30 21:48:45.074508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.073822 4526 scope.go:122] "RemoveContainer" containerID="61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279" Oct 30 21:48:45.074508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.073712 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd" Oct 30 21:48:45.087184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.087160 4526 scope.go:122] "RemoveContainer" containerID="9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa" Oct 30 21:48:45.097628 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.097600 4526 scope.go:122] "RemoveContainer" containerID="61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279" Oct 30 21:48:45.097971 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:48:45.097948 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279\": container with ID starting with 61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279 not found: ID does not exist" containerID="61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279" Oct 30 21:48:45.098042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.097981 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279"} err="failed to get container status \"61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279\": rpc error: code = NotFound desc = could not find container \"61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279\": container with ID starting with 61f19c756747ed0894583322952c073d2959e9592098ee8a7b9b1969e835e279 not found: ID does not exist" Oct 30 21:48:45.098042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.098001 4526 scope.go:122] "RemoveContainer" containerID="9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa" Oct 30 21:48:45.098277 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:48:45.098259 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa\": container with ID starting with 9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa not found: ID does not exist" containerID="9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa" Oct 30 21:48:45.098334 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.098288 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa"} err="failed to get container status \"9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa\": rpc error: code = NotFound desc = could not find container \"9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa\": container with ID starting with 9946fb9ca8c39b4a1656ef54d36aaf4f0d82017629b50502d2d0193c9760b6aa not found: ID does not exist" Oct 30 21:48:45.243507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.242535 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd"] Oct 30 21:48:45.289691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.289660 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29b7ead1-openshift-default-866d97fb49-k9hnd"] Oct 30 21:48:45.987753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:45.987721 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" path="/var/lib/kubelet/pods/0cc8fdad-213d-4748-94f3-f4c1359c8bbc/volumes" Oct 30 21:48:49.437185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.437149 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9"] Oct 30 21:48:49.437929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.437913 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" Oct 30 21:48:49.437976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.437930 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" Oct 30 21:48:49.438087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.438078 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" Oct 30 21:48:49.438120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.438094 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" Oct 30 21:48:49.443187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.443169 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:49.501209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.501164 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9"] Oct 30 21:48:49.585128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.585094 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-013fb2\"" Oct 30 21:48:49.585686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.585663 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-013fb2-dockercfg-rt45n\"" Oct 30 21:48:49.586081 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.586045 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9\" (UID: \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:49.586236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.586208 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scg24\" (UniqueName: \"kubernetes.io/projected/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-kube-api-access-scg24\") pod \"payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9\" (UID: \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:49.640067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.640033 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl"] Oct 30 21:48:49.640851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.640826 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" Oct 30 21:48:49.640851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.640847 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc8fdad-213d-4748-94f3-f4c1359c8bbc" containerName="istio-proxy" Oct 30 21:48:49.647761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.647740 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:49.680536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.680098 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl"] Oct 30 21:48:49.688241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.688164 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9\" (UID: \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:49.688367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.688292 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scg24\" (UniqueName: \"kubernetes.io/projected/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-kube-api-access-scg24\") pod \"payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9\" (UID: \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:49.688367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.688356 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2820f13c-a842-4d97-94ab-6cf0198c8cc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl\" (UID: \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:49.688548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.688529 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxl7\" (UniqueName: \"kubernetes.io/projected/2820f13c-a842-4d97-94ab-6cf0198c8cc1-kube-api-access-hwxl7\") pod \"payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl\" (UID: \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:49.689743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.689713 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9\" (UID: \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:49.746010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.745980 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scg24\" (UniqueName: \"kubernetes.io/projected/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-kube-api-access-scg24\") pod \"payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9\" (UID: \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:49.770681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.770655 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:49.789881 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.789856 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2820f13c-a842-4d97-94ab-6cf0198c8cc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl\" (UID: \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:49.790017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.789975 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxl7\" (UniqueName: \"kubernetes.io/projected/2820f13c-a842-4d97-94ab-6cf0198c8cc1-kube-api-access-hwxl7\") pod \"payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl\" (UID: \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:49.790550 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.790531 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2820f13c-a842-4d97-94ab-6cf0198c8cc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl\" (UID: \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:49.847177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.847151 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxl7\" (UniqueName: \"kubernetes.io/projected/2820f13c-a842-4d97-94ab-6cf0198c8cc1-kube-api-access-hwxl7\") pod \"payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl\" (UID: \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:49.989727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:49.989658 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:50.160028 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:50.159998 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9"] Oct 30 21:48:50.160580 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:48:50.160558 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aa1bb4_7e94_4af0_b9ec_e1c24729a7df.slice/crio-2b5f16b681d3c47c4f68a79ccc0f6ed78a1c03854d6ad65090aaf0e878fba2fc WatchSource:0}: Error finding container 2b5f16b681d3c47c4f68a79ccc0f6ed78a1c03854d6ad65090aaf0e878fba2fc: Status 404 returned error can't find the container with id 2b5f16b681d3c47c4f68a79ccc0f6ed78a1c03854d6ad65090aaf0e878fba2fc Oct 30 21:48:50.381700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:50.381672 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl"] Oct 30 21:48:50.382225 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:48:50.382205 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2820f13c_a842_4d97_94ab_6cf0198c8cc1.slice/crio-371794bab5655c07c93da5c9efe51e9c21bb6c634693376170c8413865057b59 WatchSource:0}: Error finding container 371794bab5655c07c93da5c9efe51e9c21bb6c634693376170c8413865057b59: Status 404 returned error can't find the container with id 371794bab5655c07c93da5c9efe51e9c21bb6c634693376170c8413865057b59 Oct 30 21:48:51.139188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.139155 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" event={"ID":"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df","Type":"ContainerStarted","Data":"e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a"} Oct 30 21:48:51.139706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.139212 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" event={"ID":"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df","Type":"ContainerStarted","Data":"2b5f16b681d3c47c4f68a79ccc0f6ed78a1c03854d6ad65090aaf0e878fba2fc"} Oct 30 21:48:51.139706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.139244 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:48:51.141049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.141024 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" event={"ID":"2820f13c-a842-4d97-94ab-6cf0198c8cc1","Type":"ContainerStarted","Data":"f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31"} Oct 30 21:48:51.141178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.141064 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" event={"ID":"2820f13c-a842-4d97-94ab-6cf0198c8cc1","Type":"ContainerStarted","Data":"371794bab5655c07c93da5c9efe51e9c21bb6c634693376170c8413865057b59"} Oct 30 21:48:51.141178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.141111 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:48:51.255629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.255561 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" podStartSLOduration=2.255541643 podStartE2EDuration="2.255541643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:51.251039766 +0000 UTC m=+1921.384412963" watchObservedRunningTime="2026-10-30 21:48:51.255541643 +0000 UTC m=+1921.388914840" Oct 30 21:48:51.372858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.372803 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" podStartSLOduration=2.372788633 podStartE2EDuration="2.372788633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:51.362986029 +0000 UTC m=+1921.496359228" watchObservedRunningTime="2026-10-30 21:48:51.372788633 +0000 UTC m=+1921.506161866" Oct 30 21:48:51.543836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.543807 4526 scope.go:122] "RemoveContainer" containerID="efe47ec719b3c6a0175829c3cd49d7bad5fd5b361d96e45892e54d8e75c7cb52" Oct 30 21:48:51.555597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.555575 4526 scope.go:122] "RemoveContainer" containerID="8b8a367c40c33dea0481562e9f883f978310273e241f03b97e526e2b3fc28c70" Oct 30 21:48:51.568592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.568567 4526 scope.go:122] "RemoveContainer" containerID="0b037ed0b40d940a714d3aa685213b7be86834996a19c71097946f2af9d51df3" Oct 30 21:48:51.578693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:51.578673 4526 scope.go:122] "RemoveContainer" containerID="7fc8a63e84c684deddcd4f75670bc95afd17641e50fd9806e1334671faa7fdba" Oct 30 21:48:54.364514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.363517 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w"] Oct 30 21:48:54.400564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.399250 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:54.412893 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.412827 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w"] Oct 30 21:48:54.454094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.454061 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50b188\"" Oct 30 21:48:54.454306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.454170 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50b188-dockercfg-sw29z\"" Oct 30 21:48:54.469463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.469418 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w\" (UID: \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:54.469903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.469880 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5sp5\" (UniqueName: \"kubernetes.io/projected/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-kube-api-access-k5sp5\") pod \"payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w\" (UID: \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:54.572186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.572149 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w\" (UID: \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:54.572365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.572240 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5sp5\" (UniqueName: \"kubernetes.io/projected/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-kube-api-access-k5sp5\") pod \"payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w\" (UID: \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:54.572942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.572922 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w\" (UID: \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:54.628945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.628864 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5sp5\" (UniqueName: \"kubernetes.io/projected/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-kube-api-access-k5sp5\") pod \"payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w\" (UID: \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:54.727660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:54.727624 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:55.077295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:55.077265 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:48:55.077519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:55.077335 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:48:55.107189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:55.107161 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w"] Oct 30 21:48:55.107451 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:48:55.107424 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abd0b59_18f4_4d91_b251_6c2689c8a3a4.slice/crio-034ce1e60182de719ba819a4ac478626eeca6613d1df42c26c1a62218f07a054 WatchSource:0}: Error finding container 034ce1e60182de719ba819a4ac478626eeca6613d1df42c26c1a62218f07a054: Status 404 returned error can't find the container with id 034ce1e60182de719ba819a4ac478626eeca6613d1df42c26c1a62218f07a054 Oct 30 21:48:55.204853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:55.204822 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" event={"ID":"8abd0b59-18f4-4d91-b251-6c2689c8a3a4","Type":"ContainerStarted","Data":"034ce1e60182de719ba819a4ac478626eeca6613d1df42c26c1a62218f07a054"} Oct 30 21:48:56.214191 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:56.214154 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" event={"ID":"8abd0b59-18f4-4d91-b251-6c2689c8a3a4","Type":"ContainerStarted","Data":"05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b"} Oct 30 21:48:56.214600 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:56.214244 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:48:56.329690 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:48:56.329640 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" podStartSLOduration=2.329625625 podStartE2EDuration="2.329625625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:56.32590548 +0000 UTC m=+1926.459278682" watchObservedRunningTime="2026-10-30 21:48:56.329625625 +0000 UTC m=+1926.462998823" Oct 30 21:49:01.363461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.363376 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2"] Oct 30 21:49:01.368745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.368720 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.420616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.420585 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d7f4c7-openshift-default-dockercfg-mxwhj\"" Oct 30 21:49:01.508439 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.508400 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-podinfo\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.508790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.508769 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-envoy\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.508972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.508957 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-credential-socket\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.509070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.509059 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istiod-ca-cert\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.509209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.509199 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-data\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.509340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.509330 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-certs\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.509418 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.509406 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-token\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.509578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.509563 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbcq\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-kube-api-access-5vbcq\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.509734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.509721 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-socket\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.511050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.511030 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2"] Oct 30 21:49:01.611309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611281 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-credential-socket\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.611467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611320 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istiod-ca-cert\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.611467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611388 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-data\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.611577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611453 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-certs\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.611636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611567 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-token\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.611789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611710 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbcq\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-kube-api-access-5vbcq\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.611833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611812 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-socket\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.611878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611835 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-credential-socket\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.611928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.611907 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-podinfo\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.612026 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.612011 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-envoy\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.612130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.612108 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-socket\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.612233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.612214 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istiod-ca-cert\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.612529 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.612459 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-certs\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.613018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.612994 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-data\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.614875 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.614820 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-envoy\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.614962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.614945 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-podinfo\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.670262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.670234 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-token\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.670417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.670297 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbcq\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-kube-api-access-5vbcq\") pod \"e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:01.685414 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:01.685387 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:02.132239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.132214 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2"] Oct 30 21:49:02.132935 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:49:02.132915 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6093f34_95cd_453d_bc0e_8f7bc5ecfcec.slice/crio-c85851017e6723382c6f8760aba71b77b0ba8110bf40fb2d193c91d2589610aa WatchSource:0}: Error finding container c85851017e6723382c6f8760aba71b77b0ba8110bf40fb2d193c91d2589610aa: Status 404 returned error can't find the container with id c85851017e6723382c6f8760aba71b77b0ba8110bf40fb2d193c91d2589610aa Oct 30 21:49:02.135806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.135773 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:02.135870 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.135843 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:02.135909 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.135878 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:02.153751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.153729 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:49:02.153866 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.153794 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:49:02.265721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.265680 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" event={"ID":"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec","Type":"ContainerStarted","Data":"a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13"} Oct 30 21:49:02.265880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.265742 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" event={"ID":"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec","Type":"ContainerStarted","Data":"c85851017e6723382c6f8760aba71b77b0ba8110bf40fb2d193c91d2589610aa"} Oct 30 21:49:02.507017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.506907 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podStartSLOduration=1.506891105 podStartE2EDuration="1.506891105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:02.50218535 +0000 UTC m=+1932.635558561" watchObservedRunningTime="2026-10-30 21:49:02.506891105 +0000 UTC m=+1932.640264303" Oct 30 21:49:02.686335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.686299 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:02.849623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.849562 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48"] Oct 30 21:49:02.859567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.859540 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.948921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.948885 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48"] Oct 30 21:49:02.961812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.961779 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecb65c8-f340-468e-9d6a-f731af94fd50-istiod-ca-cert\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.961989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.961872 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-certs\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.961989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.961978 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-socket\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.962110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.962022 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-podinfo\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.962110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.962055 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-data\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.962223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.962122 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-credential-socket\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.962223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.962158 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-envoy\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.962223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.962201 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4hw\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-kube-api-access-8n4hw\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.962376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.962277 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-token\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:02.996514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:02.996409 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1dc70e-openshift-default-dockercfg-l8z87\"" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.064259 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecb65c8-f340-468e-9d6a-f731af94fd50-istiod-ca-cert\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.064446 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-certs\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.064606 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-socket\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.064664 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-podinfo\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.064694 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-data\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.064762 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-credential-socket\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.064801 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-envoy\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.064858 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4hw\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-kube-api-access-8n4hw\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.067508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.067041 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-token\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.072506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.068842 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecb65c8-f340-468e-9d6a-f731af94fd50-istiod-ca-cert\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.072506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.069089 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-certs\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.072506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.069205 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-socket\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.072506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.070270 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-credential-socket\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.072506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.071324 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-data\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.073790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.073740 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-podinfo\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.076610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.076587 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-envoy\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.124195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.124120 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-token\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.126621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.126593 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4hw\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-kube-api-access-8n4hw\") pod \"e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.201565 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.201529 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:03.308040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.308002 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm"] Oct 30 21:49:03.314615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.314590 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.367021 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.366618 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:49:03.367344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.367315 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:49:03.367430 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.367397 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bckcj\"" Oct 30 21:49:03.367536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.367441 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:49:03.370595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.370567 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm"] Oct 30 21:49:03.473149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.473078 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.473149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.473119 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.473149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.473139 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.473366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.473194 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm4ql\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-kube-api-access-dm4ql\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.473366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.473333 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eda2ad-e32a-46e1-ae61-85d9a810f17c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.473543 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.473525 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.473702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.473686 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.575402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.575351 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.575948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.575415 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.575948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.575444 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.575948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.575468 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm4ql\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-kube-api-access-dm4ql\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.575948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.575529 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eda2ad-e32a-46e1-ae61-85d9a810f17c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.575948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.575712 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.575948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.575891 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.577027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.576998 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.578758 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.578735 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.578923 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.578902 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.578974 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.578919 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.579212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.579187 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eda2ad-e32a-46e1-ae61-85d9a810f17c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.633113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.633080 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.633286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.633263 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm4ql\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-kube-api-access-dm4ql\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kk6pm\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.646390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.646368 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:03.650298 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:49:03.650269 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecb65c8_f340_468e_9d6a_f731af94fd50.slice/crio-e9639024f42daeb7ac7a810237be8d859ca03611123682a3434c388d8a5e6954 WatchSource:0}: Error finding container e9639024f42daeb7ac7a810237be8d859ca03611123682a3434c388d8a5e6954: Status 404 returned error can't find the container with id e9639024f42daeb7ac7a810237be8d859ca03611123682a3434c388d8a5e6954 Oct 30 21:49:03.652747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.652718 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48"] Oct 30 21:49:03.659296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.659258 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.659371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.659346 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.659416 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.659395 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.692795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:03.692763 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:04.040613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:04.040584 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm"] Oct 30 21:49:04.040854 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:49:04.040743 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52eda2ad_e32a_46e1_ae61_85d9a810f17c.slice/crio-4eb8a4f14613f9708549c8861f836291f9a73f120c18dad69839e997c957f43e WatchSource:0}: Error finding container 4eb8a4f14613f9708549c8861f836291f9a73f120c18dad69839e997c957f43e: Status 404 returned error can't find the container with id 4eb8a4f14613f9708549c8861f836291f9a73f120c18dad69839e997c957f43e Oct 30 21:49:04.295400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:04.295305 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" event={"ID":"52eda2ad-e32a-46e1-ae61-85d9a810f17c","Type":"ContainerStarted","Data":"4eb8a4f14613f9708549c8861f836291f9a73f120c18dad69839e997c957f43e"} Oct 30 21:49:04.297106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:04.297081 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" event={"ID":"fecb65c8-f340-468e-9d6a-f731af94fd50","Type":"ContainerStarted","Data":"52f6dad2cc18b898395aafd8c16600831b331d238c90e2bcbacee62a8d3bfdb6"} Oct 30 21:49:04.297176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:04.297132 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" event={"ID":"fecb65c8-f340-468e-9d6a-f731af94fd50","Type":"ContainerStarted","Data":"e9639024f42daeb7ac7a810237be8d859ca03611123682a3434c388d8a5e6954"} Oct 30 21:49:04.297424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:04.297400 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:04.298379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:04.298361 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:49:04.419162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:04.419101 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podStartSLOduration=2.419080943 podStartE2EDuration="2.419080943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:04.414820915 +0000 UTC m=+1934.548194110" watchObservedRunningTime="2026-10-30 21:49:04.419080943 +0000 UTC m=+1934.552454143" Oct 30 21:49:05.201881 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:05.201845 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:05.207900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:05.207871 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:05.306502 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:05.306450 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:05.307437 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:05.307415 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:49:07.226565 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:07.226152 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:49:08.413424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:08.413388 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:08.413768 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:08.413459 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:09.580851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:09.580817 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" event={"ID":"52eda2ad-e32a-46e1-ae61-85d9a810f17c","Type":"ContainerStarted","Data":"585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061"} Oct 30 21:49:09.581732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:09.581700 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:09.584422 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:09.584370 4526 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kk6pm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:09.584656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:09.584622 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" podUID="52eda2ad-e32a-46e1-ae61-85d9a810f17c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:09.795523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:09.793475 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" podStartSLOduration=2.424564701 podStartE2EDuration="6.793452139s" totalImagesPullingTime="4.368887438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:09.787600333 +0000 UTC m=+1939.920973534" watchObservedRunningTime="2026-10-30 21:49:09.793452139 +0000 UTC m=+1939.926825343" Oct 30 21:49:10.593241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:10.593212 4526 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kk6pm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:10.593803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:10.593272 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" podUID="52eda2ad-e32a-46e1-ae61-85d9a810f17c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:12.480361 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:49:12.480323 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:49:13.593846 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:13.593812 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:49:18.701176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:18.701093 4526 generic.go:356] "Generic (PLEG): container finished" podID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerID="0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61" exitCode=0 Oct 30 21:49:18.701619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:18.701175 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" event={"ID":"0121879d-c1e6-4baa-91cb-c6c492fd2317","Type":"ContainerDied","Data":"0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61"} Oct 30 21:49:18.701868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:18.701849 4526 scope.go:122] "RemoveContainer" containerID="0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61" Oct 30 21:49:18.703801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:18.703771 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:18.703935 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:18.703893 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:18.704061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:18.703942 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:19.724995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:19.724957 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" event={"ID":"0121879d-c1e6-4baa-91cb-c6c492fd2317","Type":"ContainerStarted","Data":"e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507"} Oct 30 21:49:20.255813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:20.255760 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:49:20.256021 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:20.255973 4526 patch_prober.go:29] interesting pod/e2e-api-a-b34958-openshift-default-559d74f555-htq4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:20.256084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:20.256024 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:21.255526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:21.255471 4526 patch_prober.go:29] interesting pod/e2e-api-a-b34958-openshift-default-559d74f555-htq4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:21.256078 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:21.255548 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:22.222304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.222273 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68"] Oct 30 21:49:22.229800 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.229780 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.256233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.256199 4526 patch_prober.go:29] interesting pod/e2e-api-a-b34958-openshift-default-559d74f555-htq4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:22.256762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.256248 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:22.257395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.257375 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68"] Oct 30 21:49:22.281755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.281719 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:22.281951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.281884 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:49:22.282042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.282013 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:49:22.282109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.282076 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:49:22.375140 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.375101 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.375398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.375347 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5td22\" (UniqueName: \"kubernetes.io/projected/f8b3346b-3d2b-42b2-9809-0e45272beb36-kube-api-access-5td22\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.375645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.375616 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.478075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.477990 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.478075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.478029 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5td22\" (UniqueName: \"kubernetes.io/projected/f8b3346b-3d2b-42b2-9809-0e45272beb36-kube-api-access-5td22\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.478309 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.478141 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.478704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.478683 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.480846 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.480821 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.537307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.537274 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5td22\" (UniqueName: \"kubernetes.io/projected/f8b3346b-3d2b-42b2-9809-0e45272beb36-kube-api-access-5td22\") pod \"maas-api-revoke-sub-06fd552c-mjt68\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.545139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.545116 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:22.955514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:22.955472 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68"] Oct 30 21:49:22.955791 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:49:22.955767 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b3346b_3d2b_42b2_9809_0e45272beb36.slice/crio-5503d72489cce879780bd2e0574bbe933046f6b1ab7bb466b2be46b86a7467a5 WatchSource:0}: Error finding container 5503d72489cce879780bd2e0574bbe933046f6b1ab7bb466b2be46b86a7467a5: Status 404 returned error can't find the container with id 5503d72489cce879780bd2e0574bbe933046f6b1ab7bb466b2be46b86a7467a5 Oct 30 21:49:23.255984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:23.255952 4526 patch_prober.go:29] interesting pod/e2e-api-a-b34958-openshift-default-559d74f555-htq4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:23.256128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:23.256008 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:23.756441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:23.756406 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" event={"ID":"f8b3346b-3d2b-42b2-9809-0e45272beb36","Type":"ContainerStarted","Data":"b0372f3b422f02ecac50e2846bf20011c68458362989fba8ae78964fe0e8f275"} Oct 30 21:49:23.756850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:23.756460 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" event={"ID":"f8b3346b-3d2b-42b2-9809-0e45272beb36","Type":"ContainerStarted","Data":"5503d72489cce879780bd2e0574bbe933046f6b1ab7bb466b2be46b86a7467a5"} Oct 30 21:49:23.875498 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:23.875434 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" podStartSLOduration=1.87542007 podStartE2EDuration="1.87542007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:23.873006798 +0000 UTC m=+1954.006379997" watchObservedRunningTime="2026-10-30 21:49:23.87542007 +0000 UTC m=+1954.008793280" Oct 30 21:49:24.256290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:24.256245 4526 patch_prober.go:29] interesting pod/e2e-api-a-b34958-openshift-default-559d74f555-htq4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 21:49:24.256511 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:24.256316 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 21:49:24.765629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:24.765596 4526 generic.go:356] "Generic (PLEG): container finished" podID="f8b3346b-3d2b-42b2-9809-0e45272beb36" containerID="b0372f3b422f02ecac50e2846bf20011c68458362989fba8ae78964fe0e8f275" exitCode=0 Oct 30 21:49:24.766168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:24.765819 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" event={"ID":"f8b3346b-3d2b-42b2-9809-0e45272beb36","Type":"ContainerDied","Data":"b0372f3b422f02ecac50e2846bf20011c68458362989fba8ae78964fe0e8f275"} Oct 30 21:49:25.260684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:25.260655 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:49:25.261201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:25.261180 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:49:25.262089 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:25.262071 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:49:25.942584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:25.942557 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:26.058049 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.058002 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-xks-ca\") pod \"f8b3346b-3d2b-42b2-9809-0e45272beb36\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " Oct 30 21:49:26.058253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.058151 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-openshift-ca\") pod \"f8b3346b-3d2b-42b2-9809-0e45272beb36\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " Oct 30 21:49:26.058346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.058320 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b3346b-3d2b-42b2-9809-0e45272beb36-kube-api-access-5td22\" (UniqueName: \"kubernetes.io/projected/f8b3346b-3d2b-42b2-9809-0e45272beb36-kube-api-access-5td22\") pod \"f8b3346b-3d2b-42b2-9809-0e45272beb36\" (UID: \"f8b3346b-3d2b-42b2-9809-0e45272beb36\") " Oct 30 21:49:26.058784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.058738 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-openshift-ca" pod "f8b3346b-3d2b-42b2-9809-0e45272beb36" (UID: "f8b3346b-3d2b-42b2-9809-0e45272beb36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:26.059622 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.059286 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.061394 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.061365 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b3346b-3d2b-42b2-9809-0e45272beb36-kube-api-access-5td22" pod "f8b3346b-3d2b-42b2-9809-0e45272beb36" (UID: "f8b3346b-3d2b-42b2-9809-0e45272beb36"). InnerVolumeSpecName "kube-api-access-5td22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:26.061475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.061417 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-xks-ca" pod "f8b3346b-3d2b-42b2-9809-0e45272beb36" (UID: "f8b3346b-3d2b-42b2-9809-0e45272beb36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:26.161009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.160976 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b3346b-3d2b-42b2-9809-0e45272beb36-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.161009 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.161003 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5td22\" (UniqueName: \"kubernetes.io/projected/f8b3346b-3d2b-42b2-9809-0e45272beb36-kube-api-access-5td22\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.783596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.783571 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" Oct 30 21:49:26.783882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.783598 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68" event={"ID":"f8b3346b-3d2b-42b2-9809-0e45272beb36","Type":"ContainerDied","Data":"5503d72489cce879780bd2e0574bbe933046f6b1ab7bb466b2be46b86a7467a5"} Oct 30 21:49:26.783973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:26.783915 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5503d72489cce879780bd2e0574bbe933046f6b1ab7bb466b2be46b86a7467a5" Oct 30 21:49:36.049053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:36.049016 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk"] Oct 30 21:49:36.098395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:36.098366 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb1ff3d-ptwnk"] Oct 30 21:49:37.987972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:49:37.987937 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591668dd-6f81-40cc-b1fd-13017a67bd72" path="/var/lib/kubelet/pods/591668dd-6f81-40cc-b1fd-13017a67bd72/volumes" Oct 30 21:49:42.479861 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:49:42.479826 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:50:12.481882 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:50:12.481837 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:50:31.990743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:31.990651 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:50:31.990743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:31.990716 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:50:35.500723 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:35.500680 4526 generic.go:356] "Generic (PLEG): container finished" podID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerID="06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544" exitCode=0 Oct 30 21:50:35.501283 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:35.500748 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" event={"ID":"c67bb07b-9ee6-412a-a850-c872cb33b1b2","Type":"ContainerDied","Data":"06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544"} Oct 30 21:50:35.501831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:35.501808 4526 scope.go:122] "RemoveContainer" containerID="06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544" Oct 30 21:50:35.504557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:35.504470 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:35.504898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:35.504845 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:35.505589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:35.504940 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:36.523911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:36.523875 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" event={"ID":"c67bb07b-9ee6-412a-a850-c872cb33b1b2","Type":"ContainerStarted","Data":"80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a"} Oct 30 21:50:37.275304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:37.275268 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:50:37.275580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:37.275555 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:50:37.275646 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:37.275607 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 21:50:38.275780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:38.275745 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:50:38.276292 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:38.275801 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 21:50:39.275102 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:39.275068 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:50:39.275283 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:39.275122 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 21:50:40.275799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:40.275767 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:50:40.276305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:40.275839 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 21:50:41.274994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:41.274954 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:50:41.275208 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:41.275047 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 21:50:42.281083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:42.281047 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:50:42.281658 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:42.281630 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:50:42.282280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:42.282260 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:50:42.480527 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:50:42.480471 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:50:51.656803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:51.656775 4526 scope.go:122] "RemoveContainer" containerID="9ead3085b9907ccff80995ec3e8cf3c0512d2cae7ec7555bb8a2ee9f0845b4d9" Oct 30 21:50:51.667330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:50:51.667308 4526 scope.go:122] "RemoveContainer" containerID="8b43f1ccc8c3f5fbd6e07ad01a658534a62a8955e75e3e96736691a9c8754c36" Oct 30 21:51:12.491551 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:51:12.485237 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:51:19.513843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:19.459341 4526 patch_prober.go:29] interesting pod/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": read tcp 10.133.0.2:58988->10.133.0.196:15021: read: connection reset by peer" start-of-body= Oct 30 21:51:19.513843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:19.459412 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": read tcp 10.133.0.2:58988->10.133.0.196:15021: read: connection reset by peer" Oct 30 21:51:20.005884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.005802 4526 generic.go:356] "Generic (PLEG): container finished" podID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerID="a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13" exitCode=0 Oct 30 21:51:20.006085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.005880 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" event={"ID":"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec","Type":"ContainerDied","Data":"a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13"} Oct 30 21:51:20.006447 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.006422 4526 scope.go:122] "RemoveContainer" containerID="a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13" Oct 30 21:51:20.007899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.007878 4526 generic.go:356] "Generic (PLEG): container finished" podID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerID="52f6dad2cc18b898395aafd8c16600831b331d238c90e2bcbacee62a8d3bfdb6" exitCode=0 Oct 30 21:51:20.007999 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.007938 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" event={"ID":"fecb65c8-f340-468e-9d6a-f731af94fd50","Type":"ContainerDied","Data":"52f6dad2cc18b898395aafd8c16600831b331d238c90e2bcbacee62a8d3bfdb6"} Oct 30 21:51:20.008276 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.008262 4526 scope.go:122] "RemoveContainer" containerID="52f6dad2cc18b898395aafd8c16600831b331d238c90e2bcbacee62a8d3bfdb6" Oct 30 21:51:20.008465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.008442 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:20.008556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.008534 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:20.008686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.008581 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:20.009859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.009836 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:20.009934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.009894 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:20.009934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.009927 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:20.306545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:20.306378 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:51:21.044694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:21.042614 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" event={"ID":"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec","Type":"ContainerStarted","Data":"33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71"} Oct 30 21:51:21.047983 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:21.047947 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" event={"ID":"fecb65c8-f340-468e-9d6a-f731af94fd50","Type":"ContainerStarted","Data":"5b26801eec8a7de1d7e405077e8e301414efde171672edbd2b195257f3d6383f"} Oct 30 21:51:21.204937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:21.204448 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:51:21.204937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:21.204609 4526 patch_prober.go:29] interesting pod/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:51:21.204937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:21.204659 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 21:51:21.686220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:21.686188 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:51:21.686464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:21.686294 4526 patch_prober.go:29] interesting pod/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 21:51:21.686464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:21.686328 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 21:51:22.216514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:22.216417 4526 patch_prober.go:29] interesting pod/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:51:22.216514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:22.216510 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 21:51:22.686221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:22.686186 4526 patch_prober.go:29] interesting pod/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 21:51:22.686434 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:22.686261 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 21:51:23.206636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:23.206598 4526 patch_prober.go:29] interesting pod/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:51:23.206948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:23.206900 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 21:51:23.686751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:23.686720 4526 patch_prober.go:29] interesting pod/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 21:51:23.687227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:23.687198 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 21:51:24.203721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:24.203687 4526 patch_prober.go:29] interesting pod/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:51:24.204023 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:24.203995 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 21:51:24.686788 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:24.686754 4526 patch_prober.go:29] interesting pod/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 21:51:24.687224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:24.686820 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 21:51:25.203521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:25.202882 4526 patch_prober.go:29] interesting pod/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:51:25.203521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:25.202943 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 21:51:25.686192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:25.686157 4526 patch_prober.go:29] interesting pod/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 30 21:51:25.686353 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:25.686211 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 30 21:51:26.250894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:26.250863 4526 patch_prober.go:29] interesting pod/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:51:26.251338 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:26.251314 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 21:51:26.689899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:26.689870 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:51:26.690350 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:26.690332 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:51:26.690923 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:26.690899 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:51:27.207351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:27.207322 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:51:27.207816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:27.207797 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:51:27.208587 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:27.208570 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:51:29.704869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:29.704834 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw"] Oct 30 21:51:29.705315 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:29.705077 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" containerID="cri-o://9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21" gracePeriod=30 Oct 30 21:51:31.491597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:31.491549 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:51:31.492008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:31.491615 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:51:35.017865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.017837 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:51:35.089946 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.089910 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-podinfo\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.089977 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-data\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090011 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-kube-api-access-g9j9w\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-kube-api-access-g9j9w\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090050 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f07d391-d13a-482e-a61e-d7689bf80c30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f07d391-d13a-482e-a61e-d7689bf80c30-istiod-ca-cert\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090104 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-credential-socket\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090217 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-socket\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090284 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-envoy\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090341 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-data" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.090552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090532 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-certs\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090582 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-credential-socket" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.090621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090592 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f07d391-d13a-482e-a61e-d7689bf80c30-istiod-ca-cert" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:35.090621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090599 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-socket" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.090749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090591 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-token\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-token\") pod \"3f07d391-d13a-482e-a61e-d7689bf80c30\" (UID: \"3f07d391-d13a-482e-a61e-d7689bf80c30\") " Oct 30 21:51:35.090801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.090780 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-certs" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.091770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.091741 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.091906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.091773 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.091906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.091787 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.091906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.091801 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f07d391-d13a-482e-a61e-d7689bf80c30-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.091906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.091817 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.092730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.092699 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-envoy" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:35.092810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.092769 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-kube-api-access-g9j9w" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "kube-api-access-g9j9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:35.092810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.092786 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-podinfo" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:35.092975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.092957 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-token" pod "3f07d391-d13a-482e-a61e-d7689bf80c30" (UID: "3f07d391-d13a-482e-a61e-d7689bf80c30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:35.161285 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.161249 4526 generic.go:356] "Generic (PLEG): container finished" podID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerID="9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21" exitCode=0 Oct 30 21:51:35.161525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.161301 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" Oct 30 21:51:35.161525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.161295 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" event={"ID":"3f07d391-d13a-482e-a61e-d7689bf80c30","Type":"ContainerDied","Data":"9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21"} Oct 30 21:51:35.161525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.161369 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw" event={"ID":"3f07d391-d13a-482e-a61e-d7689bf80c30","Type":"ContainerDied","Data":"a28f64bbfe39cce1b603dc5a9a32ca418aaa0a556366732bbbbbfe66e0334b4d"} Oct 30 21:51:35.161525 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.161400 4526 scope.go:122] "RemoveContainer" containerID="9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21" Oct 30 21:51:35.174692 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.174672 4526 scope.go:122] "RemoveContainer" containerID="9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21" Oct 30 21:51:35.175002 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:51:35.174980 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21\": container with ID starting with 9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21 not found: ID does not exist" containerID="9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21" Oct 30 21:51:35.175087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.175010 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21"} err="failed to get container status \"9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21\": rpc error: code = NotFound desc = could not find container \"9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21\": container with ID starting with 9a1bf40ec7ac0020ed07de68f7b10cd537493efe574ed946eedf7817234d7d21 not found: ID does not exist" Oct 30 21:51:35.193924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.193891 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.193924 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.193921 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.194138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.193934 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9j9w\" (UniqueName: \"kubernetes.io/projected/3f07d391-d13a-482e-a61e-d7689bf80c30-kube-api-access-g9j9w\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.194138 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.193945 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f07d391-d13a-482e-a61e-d7689bf80c30-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.353133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.353099 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw"] Oct 30 21:51:35.361361 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.361334 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-6pscw"] Oct 30 21:51:35.987116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:35.987085 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" path="/var/lib/kubelet/pods/3f07d391-d13a-482e-a61e-d7689bf80c30/volumes" Oct 30 21:51:37.745886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.745848 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696"] Oct 30 21:51:37.746876 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.746849 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f8b3346b-3d2b-42b2-9809-0e45272beb36" containerName="revoke-keys" Oct 30 21:51:37.746876 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.746875 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b3346b-3d2b-42b2-9809-0e45272beb36" containerName="revoke-keys" Oct 30 21:51:37.747074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.746925 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" Oct 30 21:51:37.747074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.746933 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" Oct 30 21:51:37.747177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.747161 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f07d391-d13a-482e-a61e-d7689bf80c30" containerName="istio-proxy" Oct 30 21:51:37.747233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.747196 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b3346b-3d2b-42b2-9809-0e45272beb36" containerName="revoke-keys" Oct 30 21:51:37.750626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.750606 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:37.808736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.808708 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696"] Oct 30 21:51:37.844342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.844303 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grp2j\" (UniqueName: \"kubernetes.io/projected/305fd405-437d-4d8a-b546-bbc21c9b918b-kube-api-access-grp2j\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:37.844546 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.844508 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:37.844651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.844549 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:37.852902 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.852861 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:37.853092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.852965 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:37.853092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.852965 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:51:37.853092 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.852970 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:37.945846 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.945810 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grp2j\" (UniqueName: \"kubernetes.io/projected/305fd405-437d-4d8a-b546-bbc21c9b918b-kube-api-access-grp2j\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:37.946037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.945965 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:37.946037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.945994 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:37.946815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.946791 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:37.948748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:37.948729 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:38.003963 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:38.003889 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grp2j\" (UniqueName: \"kubernetes.io/projected/305fd405-437d-4d8a-b546-bbc21c9b918b-kube-api-access-grp2j\") pod \"maas-api-revoke-keys-e2e-api-a-4a1551-r8696\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:38.064324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:38.064291 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:38.449211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:38.449035 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696"] Oct 30 21:51:38.450634 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:51:38.450601 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305fd405_437d_4d8a_b546_bbc21c9b918b.slice/crio-b221804a5c0d88f732d530faaca5429d2a4bec5148a6c4876db2fcdae3f455e4 WatchSource:0}: Error finding container b221804a5c0d88f732d530faaca5429d2a4bec5148a6c4876db2fcdae3f455e4: Status 404 returned error can't find the container with id b221804a5c0d88f732d530faaca5429d2a4bec5148a6c4876db2fcdae3f455e4 Oct 30 21:51:39.199151 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:39.199071 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" event={"ID":"305fd405-437d-4d8a-b546-bbc21c9b918b","Type":"ContainerStarted","Data":"4b6b7ce9e3f24201d5d2ba6e19b742e2adce041b153de2a6427d694d072aef20"} Oct 30 21:51:39.199151 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:39.199121 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" event={"ID":"305fd405-437d-4d8a-b546-bbc21c9b918b","Type":"ContainerStarted","Data":"b221804a5c0d88f732d530faaca5429d2a4bec5148a6c4876db2fcdae3f455e4"} Oct 30 21:51:39.316892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:39.316834 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" podStartSLOduration=2.316820002 podStartE2EDuration="2.316820002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:39.314226636 +0000 UTC m=+2089.447599834" watchObservedRunningTime="2026-10-30 21:51:39.316820002 +0000 UTC m=+2089.450193200" Oct 30 21:51:40.208929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:40.208894 4526 generic.go:356] "Generic (PLEG): container finished" podID="305fd405-437d-4d8a-b546-bbc21c9b918b" containerID="4b6b7ce9e3f24201d5d2ba6e19b742e2adce041b153de2a6427d694d072aef20" exitCode=0 Oct 30 21:51:40.209323 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:40.208942 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" event={"ID":"305fd405-437d-4d8a-b546-bbc21c9b918b","Type":"ContainerDied","Data":"4b6b7ce9e3f24201d5d2ba6e19b742e2adce041b153de2a6427d694d072aef20"} Oct 30 21:51:41.404013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.403983 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:41.508176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.508083 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305fd405-437d-4d8a-b546-bbc21c9b918b-kube-api-access-grp2j\" (UniqueName: \"kubernetes.io/projected/305fd405-437d-4d8a-b546-bbc21c9b918b-kube-api-access-grp2j\") pod \"305fd405-437d-4d8a-b546-bbc21c9b918b\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " Oct 30 21:51:41.508176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.508129 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-xks-ca\") pod \"305fd405-437d-4d8a-b546-bbc21c9b918b\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " Oct 30 21:51:41.508398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.508192 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-openshift-ca\") pod \"305fd405-437d-4d8a-b546-bbc21c9b918b\" (UID: \"305fd405-437d-4d8a-b546-bbc21c9b918b\") " Oct 30 21:51:41.508613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.508587 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-openshift-ca" pod "305fd405-437d-4d8a-b546-bbc21c9b918b" (UID: "305fd405-437d-4d8a-b546-bbc21c9b918b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:41.509033 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.509014 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.510547 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.510527 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-xks-ca" pod "305fd405-437d-4d8a-b546-bbc21c9b918b" (UID: "305fd405-437d-4d8a-b546-bbc21c9b918b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:41.510674 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.510652 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305fd405-437d-4d8a-b546-bbc21c9b918b-kube-api-access-grp2j" pod "305fd405-437d-4d8a-b546-bbc21c9b918b" (UID: "305fd405-437d-4d8a-b546-bbc21c9b918b"). InnerVolumeSpecName "kube-api-access-grp2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:41.611111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.611068 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grp2j\" (UniqueName: \"kubernetes.io/projected/305fd405-437d-4d8a-b546-bbc21c9b918b-kube-api-access-grp2j\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:41.611111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:41.611103 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305fd405-437d-4d8a-b546-bbc21c9b918b-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:42.226280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:42.226248 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" Oct 30 21:51:42.226466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:42.226284 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696" event={"ID":"305fd405-437d-4d8a-b546-bbc21c9b918b","Type":"ContainerDied","Data":"b221804a5c0d88f732d530faaca5429d2a4bec5148a6c4876db2fcdae3f455e4"} Oct 30 21:51:42.226466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:42.226332 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b221804a5c0d88f732d530faaca5429d2a4bec5148a6c4876db2fcdae3f455e4" Oct 30 21:51:42.480772 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:51:42.480734 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:51:44.447440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.447402 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7"] Oct 30 21:51:44.448300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.448281 4526 cpu_manager.go:414] "containerMap: removing container" podUID="305fd405-437d-4d8a-b546-bbc21c9b918b" containerName="revoke-keys" Oct 30 21:51:44.448352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.448301 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305fd405-437d-4d8a-b546-bbc21c9b918b" containerName="revoke-keys" Oct 30 21:51:44.448470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.448461 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="305fd405-437d-4d8a-b546-bbc21c9b918b" containerName="revoke-keys" Oct 30 21:51:44.450974 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.450959 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.509680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.509648 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7"] Oct 30 21:51:44.552737 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.552691 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:44.552737 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.552713 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:44.552994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.552853 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:44.552994 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.552904 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:51:44.554813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.554792 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.555040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.555022 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn84v\" (UniqueName: \"kubernetes.io/projected/7eac36ff-950e-438f-8754-0ee213ef57e0-kube-api-access-bn84v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.555144 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.555130 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.657170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.657127 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.657335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.657234 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.657415 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.657399 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn84v\" (UniqueName: \"kubernetes.io/projected/7eac36ff-950e-438f-8754-0ee213ef57e0-kube-api-access-bn84v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.657893 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.657865 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.659810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.659783 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.716024 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.715952 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn84v\" (UniqueName: \"kubernetes.io/projected/7eac36ff-950e-438f-8754-0ee213ef57e0-kube-api-access-bn84v\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:44.764392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:44.764351 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:45.162839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:45.162810 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7"] Oct 30 21:51:45.163732 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:51:45.163706 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eac36ff_950e_438f_8754_0ee213ef57e0.slice/crio-406318701c0afb50f55767742b31b5e3515746e23fcf0eb100e63e0102ba151c WatchSource:0}: Error finding container 406318701c0afb50f55767742b31b5e3515746e23fcf0eb100e63e0102ba151c: Status 404 returned error can't find the container with id 406318701c0afb50f55767742b31b5e3515746e23fcf0eb100e63e0102ba151c Oct 30 21:51:45.253832 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:45.253788 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" event={"ID":"7eac36ff-950e-438f-8754-0ee213ef57e0","Type":"ContainerStarted","Data":"406318701c0afb50f55767742b31b5e3515746e23fcf0eb100e63e0102ba151c"} Oct 30 21:51:46.263561 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:46.263529 4526 generic.go:356] "Generic (PLEG): container finished" podID="7eac36ff-950e-438f-8754-0ee213ef57e0" containerID="429ccc8d74990369bdbada2b4796d39e0d8f8b4d83644cf580dbf8d93303223f" exitCode=0 Oct 30 21:51:46.263937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:46.263611 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" event={"ID":"7eac36ff-950e-438f-8754-0ee213ef57e0","Type":"ContainerDied","Data":"429ccc8d74990369bdbada2b4796d39e0d8f8b4d83644cf580dbf8d93303223f"} Oct 30 21:51:47.453067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.453042 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:47.497729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.497704 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-xks-ca\") pod \"7eac36ff-950e-438f-8754-0ee213ef57e0\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " Oct 30 21:51:47.497899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.497761 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-openshift-ca\") pod \"7eac36ff-950e-438f-8754-0ee213ef57e0\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " Oct 30 21:51:47.497899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.497853 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eac36ff-950e-438f-8754-0ee213ef57e0-kube-api-access-bn84v\" (UniqueName: \"kubernetes.io/projected/7eac36ff-950e-438f-8754-0ee213ef57e0-kube-api-access-bn84v\") pod \"7eac36ff-950e-438f-8754-0ee213ef57e0\" (UID: \"7eac36ff-950e-438f-8754-0ee213ef57e0\") " Oct 30 21:51:47.498202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.498164 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-openshift-ca" pod "7eac36ff-950e-438f-8754-0ee213ef57e0" (UID: "7eac36ff-950e-438f-8754-0ee213ef57e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:47.498740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.498718 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:47.500166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.500144 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-xks-ca" pod "7eac36ff-950e-438f-8754-0ee213ef57e0" (UID: "7eac36ff-950e-438f-8754-0ee213ef57e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:47.500320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.500304 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eac36ff-950e-438f-8754-0ee213ef57e0-kube-api-access-bn84v" pod "7eac36ff-950e-438f-8754-0ee213ef57e0" (UID: "7eac36ff-950e-438f-8754-0ee213ef57e0"). InnerVolumeSpecName "kube-api-access-bn84v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:47.599905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.599870 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eac36ff-950e-438f-8754-0ee213ef57e0-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:47.599905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:47.599900 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn84v\" (UniqueName: \"kubernetes.io/projected/7eac36ff-950e-438f-8754-0ee213ef57e0-kube-api-access-bn84v\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:48.283357 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:48.283323 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" Oct 30 21:51:48.283578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:48.283349 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7" event={"ID":"7eac36ff-950e-438f-8754-0ee213ef57e0","Type":"ContainerDied","Data":"406318701c0afb50f55767742b31b5e3515746e23fcf0eb100e63e0102ba151c"} Oct 30 21:51:48.283578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:48.283407 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="406318701c0afb50f55767742b31b5e3515746e23fcf0eb100e63e0102ba151c" Oct 30 21:51:49.526627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.526591 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k"] Oct 30 21:51:49.527349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.527335 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7eac36ff-950e-438f-8754-0ee213ef57e0" containerName="revoke-keys" Oct 30 21:51:49.527397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.527350 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eac36ff-950e-438f-8754-0ee213ef57e0" containerName="revoke-keys" Oct 30 21:51:49.527522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.527508 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eac36ff-950e-438f-8754-0ee213ef57e0" containerName="revoke-keys" Oct 30 21:51:49.532784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.532758 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.539752 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.539715 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtcdx\" (UniqueName: \"kubernetes.io/projected/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-kube-api-access-rtcdx\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.540006 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.539985 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.540535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.540509 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.588343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.588315 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k"] Oct 30 21:51:49.631405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.631375 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:49.631595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.631441 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:51:49.631595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.631455 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:51:49.634895 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.634873 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:51:49.642007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.641984 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.642241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.642221 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.642359 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.642343 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtcdx\" (UniqueName: \"kubernetes.io/projected/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-kube-api-access-rtcdx\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.642843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.642824 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.644569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.644550 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.699700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.699664 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtcdx\" (UniqueName: \"kubernetes.io/projected/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-kube-api-access-rtcdx\") pod \"maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:49.848478 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:49.848450 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:50.232657 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:51:50.232608 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bf16b7_fa6e_45a6_9ff6_dae6d1924c7b.slice/crio-561eb0be454229d69c68b7f12edcc46436ae3d9f4df56da363fa26d26e38eb69 WatchSource:0}: Error finding container 561eb0be454229d69c68b7f12edcc46436ae3d9f4df56da363fa26d26e38eb69: Status 404 returned error can't find the container with id 561eb0be454229d69c68b7f12edcc46436ae3d9f4df56da363fa26d26e38eb69 Oct 30 21:51:50.235944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:50.235877 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k"] Oct 30 21:51:50.302166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:50.300634 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" event={"ID":"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b","Type":"ContainerStarted","Data":"561eb0be454229d69c68b7f12edcc46436ae3d9f4df56da363fa26d26e38eb69"} Oct 30 21:51:51.311136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:51.311101 4526 generic.go:356] "Generic (PLEG): container finished" podID="e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b" containerID="4d675e347adbdcb5b00870bf253b9e5fc73ff1b4fcf5307591d69079516fb09e" exitCode=0 Oct 30 21:51:51.311700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:51.311208 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" event={"ID":"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b","Type":"ContainerDied","Data":"4d675e347adbdcb5b00870bf253b9e5fc73ff1b4fcf5307591d69079516fb09e"} Oct 30 21:51:51.598370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:51.598338 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc"] Oct 30 21:51:51.703325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:51.703288 4526 scope.go:122] "RemoveContainer" containerID="5f18bc40369200ea59730b206d1f9b1a0268d29dbf540ce089e53fc4c9935c25" Oct 30 21:51:51.714180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:51.714146 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg"] Oct 30 21:51:52.491858 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.491833 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:52.599454 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.599377 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-kube-api-access-rtcdx\" (UniqueName: \"kubernetes.io/projected/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-kube-api-access-rtcdx\") pod \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " Oct 30 21:51:52.599638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.599595 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-xks-ca\") pod \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " Oct 30 21:51:52.599701 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.599637 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-openshift-ca\") pod \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\" (UID: \"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b\") " Oct 30 21:51:52.600098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.600034 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-openshift-ca" pod "e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b" (UID: "e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:52.600304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.600288 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:52.601745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.601718 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-kube-api-access-rtcdx" pod "e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b" (UID: "e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b"). InnerVolumeSpecName "kube-api-access-rtcdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:52.601859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.601785 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-xks-ca" pod "e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b" (UID: "e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:52.701919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.701887 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtcdx\" (UniqueName: \"kubernetes.io/projected/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-kube-api-access-rtcdx\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:52.701919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:52.701917 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:53.331676 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:53.331637 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" event={"ID":"e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b","Type":"ContainerDied","Data":"561eb0be454229d69c68b7f12edcc46436ae3d9f4df56da363fa26d26e38eb69"} Oct 30 21:51:53.331889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:53.331698 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k" Oct 30 21:51:53.334542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:53.331703 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561eb0be454229d69c68b7f12edcc46436ae3d9f4df56da363fa26d26e38eb69" Oct 30 21:51:53.418604 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:51:53.418573 4526 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bf16b7_fa6e_45a6_9ff6_dae6d1924c7b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:51:56.368795 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:56.368763 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9"] Oct 30 21:51:56.478814 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:56.478777 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg"] Oct 30 21:51:56.599541 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:56.599468 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" podUID="0bae8ebc-d123-4e75-9e1a-1b2b343a70d1" containerName="payload-processing" containerID="cri-o://39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16" gracePeriod=25 Oct 30 21:51:56.715536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:56.715436 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" podUID="62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5" containerName="payload-pre-processing" containerID="cri-o://c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05" gracePeriod=25 Oct 30 21:51:56.982786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:56.982721 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:51:57.074362 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.074332 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-kube-api-access-2dbbv\" (UniqueName: \"kubernetes.io/projected/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-kube-api-access-2dbbv\") pod \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\" (UID: \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\") " Oct 30 21:51:57.074644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.074622 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-plugins-config-volume\") pod \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\" (UID: \"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1\") " Oct 30 21:51:57.074951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.074928 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-plugins-config-volume" pod "0bae8ebc-d123-4e75-9e1a-1b2b343a70d1" (UID: "0bae8ebc-d123-4e75-9e1a-1b2b343a70d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:57.075678 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.075656 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.076756 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.076738 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-kube-api-access-2dbbv" pod "0bae8ebc-d123-4e75-9e1a-1b2b343a70d1" (UID: "0bae8ebc-d123-4e75-9e1a-1b2b343a70d1"). InnerVolumeSpecName "kube-api-access-2dbbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.086470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.086450 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:51:57.176246 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.176163 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-plugins-config-volume\") pod \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\" (UID: \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\") " Oct 30 21:51:57.176405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.176368 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-kube-api-access-s8flq\" (UniqueName: \"kubernetes.io/projected/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-kube-api-access-s8flq\") pod \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\" (UID: \"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5\") " Oct 30 21:51:57.176602 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.176580 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-plugins-config-volume" pod "62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5" (UID: "62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:57.176922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.176891 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.176922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.176918 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dbbv\" (UniqueName: \"kubernetes.io/projected/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1-kube-api-access-2dbbv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.178693 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.178668 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-kube-api-access-s8flq" pod "62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5" (UID: "62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5"). InnerVolumeSpecName "kube-api-access-s8flq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:57.277883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.277849 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8flq\" (UniqueName: \"kubernetes.io/projected/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5-kube-api-access-s8flq\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:57.366527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.366477 4526 generic.go:356] "Generic (PLEG): container finished" podID="0bae8ebc-d123-4e75-9e1a-1b2b343a70d1" containerID="39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16" exitCode=0 Oct 30 21:51:57.366527 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.366518 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" event={"ID":"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1","Type":"ContainerDied","Data":"39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16"} Oct 30 21:51:57.366765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.366533 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" Oct 30 21:51:57.366765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.366581 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc" event={"ID":"0bae8ebc-d123-4e75-9e1a-1b2b343a70d1","Type":"ContainerDied","Data":"e58eda49dc0d063c95af84b1c1d6aa654ca85740328d807f88f6d07487a79911"} Oct 30 21:51:57.366765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.366610 4526 scope.go:122] "RemoveContainer" containerID="39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16" Oct 30 21:51:57.368232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.368207 4526 generic.go:356] "Generic (PLEG): container finished" podID="62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5" containerID="c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05" exitCode=0 Oct 30 21:51:57.368337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.368235 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" event={"ID":"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5","Type":"ContainerDied","Data":"c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05"} Oct 30 21:51:57.368337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.368265 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" Oct 30 21:51:57.368337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.368284 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg" event={"ID":"62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5","Type":"ContainerDied","Data":"47a82f410d32bd30cc359953b10a267500eeac0ba1df7d04807e5192f4f8ae08"} Oct 30 21:51:57.378312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.378292 4526 scope.go:122] "RemoveContainer" containerID="39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16" Oct 30 21:51:57.378674 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:51:57.378642 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16\": container with ID starting with 39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16 not found: ID does not exist" containerID="39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16" Oct 30 21:51:57.378720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.378674 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16"} err="failed to get container status \"39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16\": rpc error: code = NotFound desc = could not find container \"39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16\": container with ID starting with 39dcb5358bb8d0fc4c1e2d46ab7c90d18f7995f2a145dad4fb4e74bcc6f9ab16 not found: ID does not exist" Oct 30 21:51:57.378720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.378692 4526 scope.go:122] "RemoveContainer" containerID="c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05" Oct 30 21:51:57.389556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.389541 4526 scope.go:122] "RemoveContainer" containerID="c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05" Oct 30 21:51:57.389824 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:51:57.389805 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05\": container with ID starting with c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05 not found: ID does not exist" containerID="c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05" Oct 30 21:51:57.389890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.389835 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05"} err="failed to get container status \"c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05\": rpc error: code = NotFound desc = could not find container \"c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05\": container with ID starting with c23090b9a95644597ea4395ecf4ac69272afed878c43d026a406449ee00b9b05 not found: ID does not exist" Oct 30 21:51:57.531229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.531145 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg"] Oct 30 21:51:57.536977 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.536947 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4a1551-54849fdc55-6zbgg"] Oct 30 21:51:57.644503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.644460 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc"] Oct 30 21:51:57.650343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.650318 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a1551-57f4f89d9-6f6lc"] Oct 30 21:51:57.987659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.987621 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bae8ebc-d123-4e75-9e1a-1b2b343a70d1" path="/var/lib/kubelet/pods/0bae8ebc-d123-4e75-9e1a-1b2b343a70d1/volumes" Oct 30 21:51:57.988180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:57.988157 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5" path="/var/lib/kubelet/pods/62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5/volumes" Oct 30 21:51:58.842711 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:51:58.842681 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w"] Oct 30 21:52:01.369672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.369584 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" podUID="8637d261-f265-4b78-ae25-de43f0db20a6" containerName="payload-processing" containerID="cri-o://ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710" gracePeriod=25 Oct 30 21:52:01.479376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.479330 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" podUID="98a1afd7-9add-425e-b286-04a977e12935" containerName="payload-pre-processing" containerID="cri-o://87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5" gracePeriod=25 Oct 30 21:52:01.658296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.658267 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:52:01.764770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.764742 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8637d261-f265-4b78-ae25-de43f0db20a6-kube-api-access-xstrg\" (UniqueName: \"kubernetes.io/projected/8637d261-f265-4b78-ae25-de43f0db20a6-kube-api-access-xstrg\") pod \"8637d261-f265-4b78-ae25-de43f0db20a6\" (UID: \"8637d261-f265-4b78-ae25-de43f0db20a6\") " Oct 30 21:52:01.764948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.764869 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8637d261-f265-4b78-ae25-de43f0db20a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8637d261-f265-4b78-ae25-de43f0db20a6-plugins-config-volume\") pod \"8637d261-f265-4b78-ae25-de43f0db20a6\" (UID: \"8637d261-f265-4b78-ae25-de43f0db20a6\") " Oct 30 21:52:01.765348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.765320 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8637d261-f265-4b78-ae25-de43f0db20a6-plugins-config-volume" pod "8637d261-f265-4b78-ae25-de43f0db20a6" (UID: "8637d261-f265-4b78-ae25-de43f0db20a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:01.765959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.765935 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8637d261-f265-4b78-ae25-de43f0db20a6-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:01.767461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.767433 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8637d261-f265-4b78-ae25-de43f0db20a6-kube-api-access-xstrg" pod "8637d261-f265-4b78-ae25-de43f0db20a6" (UID: "8637d261-f265-4b78-ae25-de43f0db20a6"). InnerVolumeSpecName "kube-api-access-xstrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:01.796148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.796117 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:52:01.866704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.866668 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a1afd7-9add-425e-b286-04a977e12935-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1afd7-9add-425e-b286-04a977e12935-plugins-config-volume\") pod \"98a1afd7-9add-425e-b286-04a977e12935\" (UID: \"98a1afd7-9add-425e-b286-04a977e12935\") " Oct 30 21:52:01.866883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.866872 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a1afd7-9add-425e-b286-04a977e12935-kube-api-access-g8cmf\" (UniqueName: \"kubernetes.io/projected/98a1afd7-9add-425e-b286-04a977e12935-kube-api-access-g8cmf\") pod \"98a1afd7-9add-425e-b286-04a977e12935\" (UID: \"98a1afd7-9add-425e-b286-04a977e12935\") " Oct 30 21:52:01.867149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.867116 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a1afd7-9add-425e-b286-04a977e12935-plugins-config-volume" pod "98a1afd7-9add-425e-b286-04a977e12935" (UID: "98a1afd7-9add-425e-b286-04a977e12935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:01.867469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.867449 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstrg\" (UniqueName: \"kubernetes.io/projected/8637d261-f265-4b78-ae25-de43f0db20a6-kube-api-access-xstrg\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:01.867559 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.867475 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98a1afd7-9add-425e-b286-04a977e12935-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:01.869168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.869147 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a1afd7-9add-425e-b286-04a977e12935-kube-api-access-g8cmf" pod "98a1afd7-9add-425e-b286-04a977e12935" (UID: "98a1afd7-9add-425e-b286-04a977e12935"). InnerVolumeSpecName "kube-api-access-g8cmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:01.969148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:01.969062 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8cmf\" (UniqueName: \"kubernetes.io/projected/98a1afd7-9add-425e-b286-04a977e12935-kube-api-access-g8cmf\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:02.433323 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.433289 4526 generic.go:356] "Generic (PLEG): container finished" podID="98a1afd7-9add-425e-b286-04a977e12935" containerID="87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5" exitCode=0 Oct 30 21:52:02.433830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.433340 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" Oct 30 21:52:02.433830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.433375 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" event={"ID":"98a1afd7-9add-425e-b286-04a977e12935","Type":"ContainerDied","Data":"87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5"} Oct 30 21:52:02.433830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.433441 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg" event={"ID":"98a1afd7-9add-425e-b286-04a977e12935","Type":"ContainerDied","Data":"56fa49fb766937433f7e6d7b9ca0b25d9ea0087c7b4704f3f171055b54e75d6a"} Oct 30 21:52:02.433830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.433475 4526 scope.go:122] "RemoveContainer" containerID="87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5" Oct 30 21:52:02.435010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.434992 4526 generic.go:356] "Generic (PLEG): container finished" podID="8637d261-f265-4b78-ae25-de43f0db20a6" containerID="ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710" exitCode=0 Oct 30 21:52:02.435098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.435037 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" Oct 30 21:52:02.435098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.435078 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" event={"ID":"8637d261-f265-4b78-ae25-de43f0db20a6","Type":"ContainerDied","Data":"ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710"} Oct 30 21:52:02.435200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.435133 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9" event={"ID":"8637d261-f265-4b78-ae25-de43f0db20a6","Type":"ContainerDied","Data":"fe2bb52dc12951e87b771a3aca5483daa92e97b75c5921118d70dd1c6e50ac6f"} Oct 30 21:52:02.444432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.444411 4526 scope.go:122] "RemoveContainer" containerID="87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5" Oct 30 21:52:02.444794 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:52:02.444774 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5\": container with ID starting with 87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5 not found: ID does not exist" containerID="87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5" Oct 30 21:52:02.444864 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.444802 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5"} err="failed to get container status \"87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5\": rpc error: code = NotFound desc = could not find container \"87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5\": container with ID starting with 87b1b0384aa6a828a2b85b58cfb40646755ee143f7c7ef8d979864fd51f4d3c5 not found: ID does not exist" Oct 30 21:52:02.444864 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.444820 4526 scope.go:122] "RemoveContainer" containerID="ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710" Oct 30 21:52:02.455501 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.455461 4526 scope.go:122] "RemoveContainer" containerID="ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710" Oct 30 21:52:02.455840 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:52:02.455814 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710\": container with ID starting with ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710 not found: ID does not exist" containerID="ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710" Oct 30 21:52:02.455921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.455851 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710"} err="failed to get container status \"ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710\": rpc error: code = NotFound desc = could not find container \"ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710\": container with ID starting with ae504f5e218224e35b70411d6aa324736bfe3612dff11520e6813d95c8dec710 not found: ID does not exist" Oct 30 21:52:02.598068 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.598036 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9"] Oct 30 21:52:02.647451 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.647369 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-218800-7cfd4b86dc-9zjv9"] Oct 30 21:52:02.707923 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.707882 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg"] Oct 30 21:52:02.712079 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:02.712060 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-218800-dcb8d4f9f-5p2bg"] Oct 30 21:52:03.843231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:03.843189 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" podUID="8abd0b59-18f4-4d91-b251-6c2689c8a3a4" containerName="payload-processing" containerID="cri-o://05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b" gracePeriod=25 Oct 30 21:52:03.987861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:03.987828 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8637d261-f265-4b78-ae25-de43f0db20a6" path="/var/lib/kubelet/pods/8637d261-f265-4b78-ae25-de43f0db20a6/volumes" Oct 30 21:52:03.988315 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:03.988301 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a1afd7-9add-425e-b286-04a977e12935" path="/var/lib/kubelet/pods/98a1afd7-9add-425e-b286-04a977e12935/volumes" Oct 30 21:52:04.123548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.123520 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:52:04.201333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.201302 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-kube-api-access-k5sp5\" (UniqueName: \"kubernetes.io/projected/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-kube-api-access-k5sp5\") pod \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\" (UID: \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\") " Oct 30 21:52:04.201554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.201509 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-plugins-config-volume\") pod \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\" (UID: \"8abd0b59-18f4-4d91-b251-6c2689c8a3a4\") " Oct 30 21:52:04.201941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.201914 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-plugins-config-volume" pod "8abd0b59-18f4-4d91-b251-6c2689c8a3a4" (UID: "8abd0b59-18f4-4d91-b251-6c2689c8a3a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:04.202925 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.202871 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:04.205352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.205323 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-kube-api-access-k5sp5" pod "8abd0b59-18f4-4d91-b251-6c2689c8a3a4" (UID: "8abd0b59-18f4-4d91-b251-6c2689c8a3a4"). InnerVolumeSpecName "kube-api-access-k5sp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:04.304906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.304871 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5sp5\" (UniqueName: \"kubernetes.io/projected/8abd0b59-18f4-4d91-b251-6c2689c8a3a4-kube-api-access-k5sp5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:04.457468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.457381 4526 generic.go:356] "Generic (PLEG): container finished" podID="8abd0b59-18f4-4d91-b251-6c2689c8a3a4" containerID="05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b" exitCode=0 Oct 30 21:52:04.457468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.457437 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" Oct 30 21:52:04.457783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.457472 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" event={"ID":"8abd0b59-18f4-4d91-b251-6c2689c8a3a4","Type":"ContainerDied","Data":"05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b"} Oct 30 21:52:04.457783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.457548 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w" event={"ID":"8abd0b59-18f4-4d91-b251-6c2689c8a3a4","Type":"ContainerDied","Data":"034ce1e60182de719ba819a4ac478626eeca6613d1df42c26c1a62218f07a054"} Oct 30 21:52:04.457783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.457572 4526 scope.go:122] "RemoveContainer" containerID="05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b" Oct 30 21:52:04.468846 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.468828 4526 scope.go:122] "RemoveContainer" containerID="05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b" Oct 30 21:52:04.469130 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:52:04.469106 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b\": container with ID starting with 05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b not found: ID does not exist" containerID="05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b" Oct 30 21:52:04.469196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.469145 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b"} err="failed to get container status \"05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b\": rpc error: code = NotFound desc = could not find container \"05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b\": container with ID starting with 05e6f3e12e609eec2f42f19e8c515c685ba21b474bec22e27e8920bc470c841b not found: ID does not exist" Oct 30 21:52:04.623304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.623221 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w"] Oct 30 21:52:04.629794 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:04.629763 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b188-9b764f45-cqn2w"] Oct 30 21:52:05.987432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:05.987398 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abd0b59-18f4-4d91-b251-6c2689c8a3a4" path="/var/lib/kubelet/pods/8abd0b59-18f4-4d91-b251-6c2689c8a3a4/volumes" Oct 30 21:52:09.228922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.228886 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4"] Oct 30 21:52:09.229673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229656 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b" containerName="revoke-keys" Oct 30 21:52:09.229720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229674 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b" containerName="revoke-keys" Oct 30 21:52:09.229720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229690 4526 cpu_manager.go:414] "containerMap: removing container" podUID="62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5" containerName="payload-pre-processing" Oct 30 21:52:09.229720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229696 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5" containerName="payload-pre-processing" Oct 30 21:52:09.229835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229722 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0bae8ebc-d123-4e75-9e1a-1b2b343a70d1" containerName="payload-processing" Oct 30 21:52:09.229835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229727 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bae8ebc-d123-4e75-9e1a-1b2b343a70d1" containerName="payload-processing" Oct 30 21:52:09.229835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229742 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8abd0b59-18f4-4d91-b251-6c2689c8a3a4" containerName="payload-processing" Oct 30 21:52:09.229835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229747 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abd0b59-18f4-4d91-b251-6c2689c8a3a4" containerName="payload-processing" Oct 30 21:52:09.229835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229760 4526 cpu_manager.go:414] "containerMap: removing container" podUID="98a1afd7-9add-425e-b286-04a977e12935" containerName="payload-pre-processing" Oct 30 21:52:09.229835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229765 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a1afd7-9add-425e-b286-04a977e12935" containerName="payload-pre-processing" Oct 30 21:52:09.229835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229779 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8637d261-f265-4b78-ae25-de43f0db20a6" containerName="payload-processing" Oct 30 21:52:09.229835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229784 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8637d261-f265-4b78-ae25-de43f0db20a6" containerName="payload-processing" Oct 30 21:52:09.230070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229884 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a1afd7-9add-425e-b286-04a977e12935" containerName="payload-pre-processing" Oct 30 21:52:09.230070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229897 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8637d261-f265-4b78-ae25-de43f0db20a6" containerName="payload-processing" Oct 30 21:52:09.230070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229905 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b" containerName="revoke-keys" Oct 30 21:52:09.230070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229912 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d5325c-bcc7-4a13-b2ac-286cfc8ae6a5" containerName="payload-pre-processing" Oct 30 21:52:09.230070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229921 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bae8ebc-d123-4e75-9e1a-1b2b343a70d1" containerName="payload-processing" Oct 30 21:52:09.230070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.229928 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abd0b59-18f4-4d91-b251-6c2689c8a3a4" containerName="payload-processing" Oct 30 21:52:09.234905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.234887 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.281047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.281006 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.281233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.281136 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr7kl\" (UniqueName: \"kubernetes.io/projected/d4b11d68-9460-470d-bae5-4528db915f54-kube-api-access-pr7kl\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.281356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.281332 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.306515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.292595 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4"] Oct 30 21:52:09.335739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.335707 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:52:09.335739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.335707 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:52:09.335971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.335707 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:52:09.343478 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.343455 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:52:09.382779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.382743 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.382977 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.382865 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr7kl\" (UniqueName: \"kubernetes.io/projected/d4b11d68-9460-470d-bae5-4528db915f54-kube-api-access-pr7kl\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.383114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.383097 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.384053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.384020 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.385645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.385624 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.440310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.440276 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr7kl\" (UniqueName: \"kubernetes.io/projected/d4b11d68-9460-470d-bae5-4528db915f54-kube-api-access-pr7kl\") pod \"maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.549253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.549206 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:09.937059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:09.937029 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4"] Oct 30 21:52:09.937704 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:52:09.937675 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b11d68_9460_470d_bae5_4528db915f54.slice/crio-30772b8bc5c01758ee711b680b439f2ea1100f4d31201d789db54baabe2e7d30 WatchSource:0}: Error finding container 30772b8bc5c01758ee711b680b439f2ea1100f4d31201d789db54baabe2e7d30: Status 404 returned error can't find the container with id 30772b8bc5c01758ee711b680b439f2ea1100f4d31201d789db54baabe2e7d30 Oct 30 21:52:10.505112 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:10.505018 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" event={"ID":"d4b11d68-9460-470d-bae5-4528db915f54","Type":"ContainerStarted","Data":"35f4ee97d810edfab9a50051f5dc1ad55459f9f04d1efdfe607e592753379a54"} Oct 30 21:52:10.505112 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:10.505073 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" event={"ID":"d4b11d68-9460-470d-bae5-4528db915f54","Type":"ContainerStarted","Data":"30772b8bc5c01758ee711b680b439f2ea1100f4d31201d789db54baabe2e7d30"} Oct 30 21:52:10.619629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:10.619567 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" podStartSLOduration=1.619552193 podStartE2EDuration="1.619552193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:10.617531447 +0000 UTC m=+2120.750904644" watchObservedRunningTime="2026-10-30 21:52:10.619552193 +0000 UTC m=+2120.752925449" Oct 30 21:52:10.933373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:10.933334 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm"] Oct 30 21:52:10.937868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:10.937844 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:10.996175 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:10.996144 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm"] Oct 30 21:52:11.008459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.008419 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.008629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.008524 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4zp\" (UniqueName: \"kubernetes.io/projected/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-kube-api-access-7l4zp\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.008673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.008610 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.110922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.110816 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.110922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.110893 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4zp\" (UniqueName: \"kubernetes.io/projected/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-kube-api-access-7l4zp\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.110922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.110911 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.111571 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.111547 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.113458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.113440 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.167266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.167229 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4zp\" (UniqueName: \"kubernetes.io/projected/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-kube-api-access-7l4zp\") pod \"maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.252436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.252403 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:11.514469 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.514380 4526 generic.go:356] "Generic (PLEG): container finished" podID="d4b11d68-9460-470d-bae5-4528db915f54" containerID="35f4ee97d810edfab9a50051f5dc1ad55459f9f04d1efdfe607e592753379a54" exitCode=0 Oct 30 21:52:11.514896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.514464 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" event={"ID":"d4b11d68-9460-470d-bae5-4528db915f54","Type":"ContainerDied","Data":"35f4ee97d810edfab9a50051f5dc1ad55459f9f04d1efdfe607e592753379a54"} Oct 30 21:52:11.645806 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:52:11.645770 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cca1b4_de1d_4ae1_91e8_032e7b9b24ed.slice/crio-de972735e541904329b2f517e2443d1184093350eb4f7d4c535abba7affe72f0 WatchSource:0}: Error finding container de972735e541904329b2f517e2443d1184093350eb4f7d4c535abba7affe72f0: Status 404 returned error can't find the container with id de972735e541904329b2f517e2443d1184093350eb4f7d4c535abba7affe72f0 Oct 30 21:52:11.682780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:11.682752 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm"] Oct 30 21:52:12.480251 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:52:12.480214 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:52:12.523979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.523945 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" event={"ID":"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed","Type":"ContainerStarted","Data":"3275c78f8a605e1490220976c5587e438ac9875bdebaafa31a34311d9c36f734"} Oct 30 21:52:12.524439 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.523995 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" event={"ID":"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed","Type":"ContainerStarted","Data":"de972735e541904329b2f517e2443d1184093350eb4f7d4c535abba7affe72f0"} Oct 30 21:52:12.694166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.693984 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:12.741650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.741565 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-openshift-ca\") pod \"d4b11d68-9460-470d-bae5-4528db915f54\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " Oct 30 21:52:12.741817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.741698 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b11d68-9460-470d-bae5-4528db915f54-kube-api-access-pr7kl\" (UniqueName: \"kubernetes.io/projected/d4b11d68-9460-470d-bae5-4528db915f54-kube-api-access-pr7kl\") pod \"d4b11d68-9460-470d-bae5-4528db915f54\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " Oct 30 21:52:12.741817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.741756 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-xks-ca\") pod \"d4b11d68-9460-470d-bae5-4528db915f54\" (UID: \"d4b11d68-9460-470d-bae5-4528db915f54\") " Oct 30 21:52:12.741987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.741966 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-openshift-ca" pod "d4b11d68-9460-470d-bae5-4528db915f54" (UID: "d4b11d68-9460-470d-bae5-4528db915f54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:12.742532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.742477 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:12.743969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.743942 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-xks-ca" pod "d4b11d68-9460-470d-bae5-4528db915f54" (UID: "d4b11d68-9460-470d-bae5-4528db915f54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:12.744058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.744038 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b11d68-9460-470d-bae5-4528db915f54-kube-api-access-pr7kl" pod "d4b11d68-9460-470d-bae5-4528db915f54" (UID: "d4b11d68-9460-470d-bae5-4528db915f54"). InnerVolumeSpecName "kube-api-access-pr7kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:12.829468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.829399 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" podStartSLOduration=2.829378939 podStartE2EDuration="2.829378939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:12.641852131 +0000 UTC m=+2122.775225330" watchObservedRunningTime="2026-10-30 21:52:12.829378939 +0000 UTC m=+2122.962752138" Oct 30 21:52:12.843884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.843851 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr7kl\" (UniqueName: \"kubernetes.io/projected/d4b11d68-9460-470d-bae5-4528db915f54-kube-api-access-pr7kl\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:12.843884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:12.843880 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b11d68-9460-470d-bae5-4528db915f54-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:13.532802 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:13.532764 4526 generic.go:356] "Generic (PLEG): container finished" podID="77cca1b4-de1d-4ae1-91e8-032e7b9b24ed" containerID="3275c78f8a605e1490220976c5587e438ac9875bdebaafa31a34311d9c36f734" exitCode=0 Oct 30 21:52:13.533195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:13.532849 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" event={"ID":"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed","Type":"ContainerDied","Data":"3275c78f8a605e1490220976c5587e438ac9875bdebaafa31a34311d9c36f734"} Oct 30 21:52:13.534374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:13.534355 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" Oct 30 21:52:13.534481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:13.534350 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4" event={"ID":"d4b11d68-9460-470d-bae5-4528db915f54","Type":"ContainerDied","Data":"30772b8bc5c01758ee711b680b439f2ea1100f4d31201d789db54baabe2e7d30"} Oct 30 21:52:13.534481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:13.534471 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30772b8bc5c01758ee711b680b439f2ea1100f4d31201d789db54baabe2e7d30" Oct 30 21:52:14.716807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.716783 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:14.775668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.775630 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-kube-api-access-7l4zp\" (UniqueName: \"kubernetes.io/projected/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-kube-api-access-7l4zp\") pod \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " Oct 30 21:52:14.775852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.775770 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-openshift-ca\") pod \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " Oct 30 21:52:14.775852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.775840 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-xks-ca\") pod \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\" (UID: \"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed\") " Oct 30 21:52:14.776210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.776177 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-openshift-ca" pod "77cca1b4-de1d-4ae1-91e8-032e7b9b24ed" (UID: "77cca1b4-de1d-4ae1-91e8-032e7b9b24ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:14.776590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.776564 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:14.778091 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.778064 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-kube-api-access-7l4zp" pod "77cca1b4-de1d-4ae1-91e8-032e7b9b24ed" (UID: "77cca1b4-de1d-4ae1-91e8-032e7b9b24ed"). InnerVolumeSpecName "kube-api-access-7l4zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:14.778517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.778465 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-xks-ca" pod "77cca1b4-de1d-4ae1-91e8-032e7b9b24ed" (UID: "77cca1b4-de1d-4ae1-91e8-032e7b9b24ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:14.878510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.878413 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:14.878510 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:14.878454 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l4zp\" (UniqueName: \"kubernetes.io/projected/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed-kube-api-access-7l4zp\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:15.551062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:15.551028 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" Oct 30 21:52:15.551269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:15.551058 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm" event={"ID":"77cca1b4-de1d-4ae1-91e8-032e7b9b24ed","Type":"ContainerDied","Data":"de972735e541904329b2f517e2443d1184093350eb4f7d4c535abba7affe72f0"} Oct 30 21:52:15.551269 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:15.551107 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de972735e541904329b2f517e2443d1184093350eb4f7d4c535abba7affe72f0" Oct 30 21:52:23.544180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:23.544144 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv"] Oct 30 21:52:23.656046 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:23.655992 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc"] Oct 30 21:52:26.050368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:26.050333 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb"] Oct 30 21:52:26.100037 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:26.100008 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49c3b3a2-wcgnb"] Oct 30 21:52:27.988442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:27.988409 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04cdcfe-02d9-4dc8-86da-2774f70f24d8" path="/var/lib/kubelet/pods/b04cdcfe-02d9-4dc8-86da-2774f70f24d8/volumes" Oct 30 21:52:28.213496 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.213451 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podUID="a4207178a950303c256d4f333088d96f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:52:28.544896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.544851 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" podUID="1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d" containerName="payload-processing" containerID="cri-o://06f8c4b1feaaee63ae844791bb104b92eb4be0257de0663f3ad723d08d4e080d" gracePeriod=25 Oct 30 21:52:28.657700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.657652 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" podUID="975eb4b3-04cd-48f5-97ff-c888f0502f39" containerName="payload-pre-processing" containerID="cri-o://99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077" gracePeriod=25 Oct 30 21:52:28.677082 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.677049 4526 generic.go:356] "Generic (PLEG): container finished" podID="1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d" containerID="06f8c4b1feaaee63ae844791bb104b92eb4be0257de0663f3ad723d08d4e080d" exitCode=0 Oct 30 21:52:28.677250 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.677141 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" event={"ID":"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d","Type":"ContainerDied","Data":"06f8c4b1feaaee63ae844791bb104b92eb4be0257de0663f3ad723d08d4e080d"} Oct 30 21:52:28.845892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.845862 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:52:28.911890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.911858 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-plugins-config-volume\") pod \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\" (UID: \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\") " Oct 30 21:52:28.912103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.912083 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-kube-api-access-bgldj\") pod \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\" (UID: \"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d\") " Oct 30 21:52:28.912244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.912218 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-plugins-config-volume" pod "1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d" (UID: "1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:28.912713 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.912695 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:28.914953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.914926 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-kube-api-access-bgldj" pod "1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d" (UID: "1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d"). InnerVolumeSpecName "kube-api-access-bgldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:28.958193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:28.958168 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:52:29.013857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.013822 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975eb4b3-04cd-48f5-97ff-c888f0502f39-kube-api-access-mmvjj\" (UniqueName: \"kubernetes.io/projected/975eb4b3-04cd-48f5-97ff-c888f0502f39-kube-api-access-mmvjj\") pod \"975eb4b3-04cd-48f5-97ff-c888f0502f39\" (UID: \"975eb4b3-04cd-48f5-97ff-c888f0502f39\") " Oct 30 21:52:29.014348 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.013970 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975eb4b3-04cd-48f5-97ff-c888f0502f39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975eb4b3-04cd-48f5-97ff-c888f0502f39-plugins-config-volume\") pod \"975eb4b3-04cd-48f5-97ff-c888f0502f39\" (UID: \"975eb4b3-04cd-48f5-97ff-c888f0502f39\") " Oct 30 21:52:29.014448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.014419 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975eb4b3-04cd-48f5-97ff-c888f0502f39-plugins-config-volume" pod "975eb4b3-04cd-48f5-97ff-c888f0502f39" (UID: "975eb4b3-04cd-48f5-97ff-c888f0502f39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:29.014824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.014798 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d-kube-api-access-bgldj\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:29.014890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.014834 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975eb4b3-04cd-48f5-97ff-c888f0502f39-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:29.016984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.016954 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975eb4b3-04cd-48f5-97ff-c888f0502f39-kube-api-access-mmvjj" pod "975eb4b3-04cd-48f5-97ff-c888f0502f39" (UID: "975eb4b3-04cd-48f5-97ff-c888f0502f39"). InnerVolumeSpecName "kube-api-access-mmvjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:29.116438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.116347 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvjj\" (UniqueName: \"kubernetes.io/projected/975eb4b3-04cd-48f5-97ff-c888f0502f39-kube-api-access-mmvjj\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:29.685900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.685869 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" Oct 30 21:52:29.685900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.685883 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv" event={"ID":"1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d","Type":"ContainerDied","Data":"3b826ba6c3460de257e38cf1507b70898c33181f4a7614e83d6c4ba8f5eeec0d"} Oct 30 21:52:29.686170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.685978 4526 scope.go:122] "RemoveContainer" containerID="06f8c4b1feaaee63ae844791bb104b92eb4be0257de0663f3ad723d08d4e080d" Oct 30 21:52:29.687708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.687681 4526 generic.go:356] "Generic (PLEG): container finished" podID="975eb4b3-04cd-48f5-97ff-c888f0502f39" containerID="99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077" exitCode=0 Oct 30 21:52:29.687836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.687745 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" Oct 30 21:52:29.687836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.687769 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" event={"ID":"975eb4b3-04cd-48f5-97ff-c888f0502f39","Type":"ContainerDied","Data":"99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077"} Oct 30 21:52:29.687836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.687820 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc" event={"ID":"975eb4b3-04cd-48f5-97ff-c888f0502f39","Type":"ContainerDied","Data":"42ec6c2fb7628f8f3c70eeb5eabff07254934ee796ab20dde57e72912eb645b0"} Oct 30 21:52:29.697477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.697453 4526 scope.go:122] "RemoveContainer" containerID="99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077" Oct 30 21:52:29.708314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.708298 4526 scope.go:122] "RemoveContainer" containerID="99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077" Oct 30 21:52:29.708597 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:52:29.708577 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077\": container with ID starting with 99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077 not found: ID does not exist" containerID="99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077" Oct 30 21:52:29.708673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.708605 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077"} err="failed to get container status \"99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077\": rpc error: code = NotFound desc = could not find container \"99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077\": container with ID starting with 99c7d388386b161a08fdbcac7693d5c2d279021c937b44f4d51b77119cd64077 not found: ID does not exist" Oct 30 21:52:29.851034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.850940 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv"] Oct 30 21:52:29.855019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.854990 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbb72e-5575659479-vz9nv"] Oct 30 21:52:29.987236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:29.987201 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d" path="/var/lib/kubelet/pods/1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d/volumes" Oct 30 21:52:30.007755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:30.007722 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc"] Oct 30 21:52:30.015457 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:30.015427 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-fbb72e-6d9c658687-q9cjc"] Oct 30 21:52:31.986717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:31.986687 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975eb4b3-04cd-48f5-97ff-c888f0502f39" path="/var/lib/kubelet/pods/975eb4b3-04cd-48f5-97ff-c888f0502f39/volumes" Oct 30 21:52:42.480771 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:52:42.480728 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:52:45.049149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:45.049103 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx"] Oct 30 21:52:45.098573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:45.098538 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c5afda-ttmjx"] Oct 30 21:52:45.988130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:52:45.988096 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc5889f-5d8f-4050-aecb-ac95b21e9ce2" path="/var/lib/kubelet/pods/7fc5889f-5d8f-4050-aecb-ac95b21e9ce2/volumes" Oct 30 21:53:12.493313 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:53:12.493273 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:53:12.817988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:12.817953 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7"] Oct 30 21:53:12.865093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:12.865061 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-218800-wnng7"] Oct 30 21:53:13.987099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:13.987068 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eac36ff-950e-438f-8754-0ee213ef57e0" path="/var/lib/kubelet/pods/7eac36ff-950e-438f-8754-0ee213ef57e0/volumes" Oct 30 21:53:21.290524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:21.290476 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd"] Oct 30 21:53:21.290969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:21.290747 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" containerID="cri-o://8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68" gracePeriod=30 Oct 30 21:53:22.104436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.104408 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb"] Oct 30 21:53:22.105152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105136 4526 cpu_manager.go:414] "containerMap: removing container" podUID="1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d" containerName="payload-processing" Oct 30 21:53:22.105213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105154 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d" containerName="payload-processing" Oct 30 21:53:22.105213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105192 4526 cpu_manager.go:414] "containerMap: removing container" podUID="d4b11d68-9460-470d-bae5-4528db915f54" containerName="revoke-keys" Oct 30 21:53:22.105213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105198 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b11d68-9460-470d-bae5-4528db915f54" containerName="revoke-keys" Oct 30 21:53:22.105213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105208 4526 cpu_manager.go:414] "containerMap: removing container" podUID="975eb4b3-04cd-48f5-97ff-c888f0502f39" containerName="payload-pre-processing" Oct 30 21:53:22.105346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105214 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975eb4b3-04cd-48f5-97ff-c888f0502f39" containerName="payload-pre-processing" Oct 30 21:53:22.105346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105228 4526 cpu_manager.go:414] "containerMap: removing container" podUID="77cca1b4-de1d-4ae1-91e8-032e7b9b24ed" containerName="revoke-keys" Oct 30 21:53:22.105346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105232 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77cca1b4-de1d-4ae1-91e8-032e7b9b24ed" containerName="revoke-keys" Oct 30 21:53:22.105346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105339 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b11d68-9460-470d-bae5-4528db915f54" containerName="revoke-keys" Oct 30 21:53:22.105470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105349 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="975eb4b3-04cd-48f5-97ff-c888f0502f39" containerName="payload-pre-processing" Oct 30 21:53:22.105470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105356 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="77cca1b4-de1d-4ae1-91e8-032e7b9b24ed" containerName="revoke-keys" Oct 30 21:53:22.105470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.105366 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2f44f4-ec6c-44b1-b0c9-5888abe8fe6d" containerName="payload-processing" Oct 30 21:53:22.108881 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.108865 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.142151 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.142121 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4d9f\" (UniqueName: \"kubernetes.io/projected/ff7d2f5d-e789-46c6-a38a-f4c960126d85-kube-api-access-j4d9f\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.142304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.142283 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.142376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.142359 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.161568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.161545 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:22.161725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.161572 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:53:22.162725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.162705 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:22.162835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.162738 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:22.215537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.215497 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb"] Oct 30 21:53:22.244324 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.244271 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4d9f\" (UniqueName: \"kubernetes.io/projected/ff7d2f5d-e789-46c6-a38a-f4c960126d85-kube-api-access-j4d9f\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.244580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.244424 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.244580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.244509 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.245164 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.245138 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.247770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.247744 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.304106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.304071 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4d9f\" (UniqueName: \"kubernetes.io/projected/ff7d2f5d-e789-46c6-a38a-f4c960126d85-kube-api-access-j4d9f\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.423710 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.423630 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:22.833560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:22.828000 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb"] Oct 30 21:53:23.137042 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:23.137007 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" event={"ID":"ff7d2f5d-e789-46c6-a38a-f4c960126d85","Type":"ContainerStarted","Data":"2f7faeefc2f1c4dbbde9f5ee8d1699a21181455870665c8687a510a5ae825eac"} Oct 30 21:53:24.146056 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:24.146019 4526 generic.go:356] "Generic (PLEG): container finished" podID="ff7d2f5d-e789-46c6-a38a-f4c960126d85" containerID="8e20cb064640947b4d68b98d101b037ca649982506f1e31a4cc7f60eae2c202f" exitCode=0 Oct 30 21:53:24.146445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:24.146108 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" event={"ID":"ff7d2f5d-e789-46c6-a38a-f4c960126d85","Type":"ContainerDied","Data":"8e20cb064640947b4d68b98d101b037ca649982506f1e31a4cc7f60eae2c202f"} Oct 30 21:53:25.025696 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.025663 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696"] Oct 30 21:53:25.074906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.074874 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a1551-r8696"] Oct 30 21:53:25.310820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.310797 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:25.404551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.404516 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-xks-ca\") pod \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " Oct 30 21:53:25.404725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.404657 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-openshift-ca\") pod \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " Oct 30 21:53:25.404725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.404690 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7d2f5d-e789-46c6-a38a-f4c960126d85-kube-api-access-j4d9f\" (UniqueName: \"kubernetes.io/projected/ff7d2f5d-e789-46c6-a38a-f4c960126d85-kube-api-access-j4d9f\") pod \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\" (UID: \"ff7d2f5d-e789-46c6-a38a-f4c960126d85\") " Oct 30 21:53:25.404996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.404973 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-openshift-ca" pod "ff7d2f5d-e789-46c6-a38a-f4c960126d85" (UID: "ff7d2f5d-e789-46c6-a38a-f4c960126d85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:25.405410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.405351 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:25.406734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.406709 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-xks-ca" pod "ff7d2f5d-e789-46c6-a38a-f4c960126d85" (UID: "ff7d2f5d-e789-46c6-a38a-f4c960126d85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:25.407035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.407012 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7d2f5d-e789-46c6-a38a-f4c960126d85-kube-api-access-j4d9f" pod "ff7d2f5d-e789-46c6-a38a-f4c960126d85" (UID: "ff7d2f5d-e789-46c6-a38a-f4c960126d85"). InnerVolumeSpecName "kube-api-access-j4d9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:25.507584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.507553 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4d9f\" (UniqueName: \"kubernetes.io/projected/ff7d2f5d-e789-46c6-a38a-f4c960126d85-kube-api-access-j4d9f\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:25.507584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.507587 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff7d2f5d-e789-46c6-a38a-f4c960126d85-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:25.986294 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:25.986262 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305fd405-437d-4d8a-b546-bbc21c9b918b" path="/var/lib/kubelet/pods/305fd405-437d-4d8a-b546-bbc21c9b918b/volumes" Oct 30 21:53:26.162761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.162727 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" event={"ID":"ff7d2f5d-e789-46c6-a38a-f4c960126d85","Type":"ContainerDied","Data":"2f7faeefc2f1c4dbbde9f5ee8d1699a21181455870665c8687a510a5ae825eac"} Oct 30 21:53:26.162942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.162777 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f7faeefc2f1c4dbbde9f5ee8d1699a21181455870665c8687a510a5ae825eac" Oct 30 21:53:26.162942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.162738 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb" Oct 30 21:53:26.571391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.571337 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:53:26.722631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.722601 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-envoy\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.722828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.722645 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-certs\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.722828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.722708 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-token\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-token\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.722828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.722750 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-credential-socket\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.722828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.722792 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-data\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.723064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.722918 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-socket\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.723064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.722946 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-kube-api-access-kqmcr\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-kube-api-access-kqmcr\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.723064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.723000 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istiod-ca-cert\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.723227 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.723075 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-podinfo\") pod \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\" (UID: \"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb\") " Oct 30 21:53:26.723881 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.722993 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-certs" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:26.724083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.723255 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-data" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:26.724201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.723434 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-credential-socket" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:26.724321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.723645 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-socket" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:26.724408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.724030 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istiod-ca-cert" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:26.725767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.725738 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-token" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:26.725985 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.725962 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-envoy" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:26.726349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.726330 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-podinfo" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:53:26.727207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.727186 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-kube-api-access-kqmcr" pod "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" (UID: "eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb"). InnerVolumeSpecName "kube-api-access-kqmcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:26.825642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825564 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:26.825642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825591 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:26.825642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825606 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqmcr\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-kube-api-access-kqmcr\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:26.825642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825619 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:26.825642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825632 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:26.826041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825645 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:26.826041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825657 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:26.826041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825669 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:26.826041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:26.825681 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:27.171718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.171626 4526 generic.go:356] "Generic (PLEG): container finished" podID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerID="8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68" exitCode=0 Oct 30 21:53:27.171718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.171687 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" Oct 30 21:53:27.171941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.171716 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" event={"ID":"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb","Type":"ContainerDied","Data":"8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68"} Oct 30 21:53:27.171941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.171778 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd" event={"ID":"eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb","Type":"ContainerDied","Data":"10e9712db5434df6008321f5e751fc4a516928bf5264707a1dd28553fdc922d9"} Oct 30 21:53:27.171941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.171803 4526 scope.go:122] "RemoveContainer" containerID="8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68" Oct 30 21:53:27.183069 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.183050 4526 scope.go:122] "RemoveContainer" containerID="0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58" Oct 30 21:53:27.192656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.192639 4526 scope.go:122] "RemoveContainer" containerID="8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68" Oct 30 21:53:27.192910 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:53:27.192892 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68\": container with ID starting with 8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68 not found: ID does not exist" containerID="8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68" Oct 30 21:53:27.192968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.192917 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68"} err="failed to get container status \"8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68\": rpc error: code = NotFound desc = could not find container \"8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68\": container with ID starting with 8795370d8cd1d2c72b8f44867db797dfe44e4063040c0a4c85281a49838f7e68 not found: ID does not exist" Oct 30 21:53:27.192968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.192937 4526 scope.go:122] "RemoveContainer" containerID="0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58" Oct 30 21:53:27.193125 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:53:27.193107 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58\": container with ID starting with 0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58 not found: ID does not exist" containerID="0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58" Oct 30 21:53:27.193160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.193134 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58"} err="failed to get container status \"0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58\": rpc error: code = NotFound desc = could not find container \"0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58\": container with ID starting with 0e5462a17bf2db943486545d4f077641cd56569efe408abfb9f9087279a4ff58 not found: ID does not exist" Oct 30 21:53:27.344588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.344556 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd"] Oct 30 21:53:27.349088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.349060 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-6zvtd"] Oct 30 21:53:27.987476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:27.987442 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" path="/var/lib/kubelet/pods/eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb/volumes" Oct 30 21:53:33.778537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:33.778426 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl"] Oct 30 21:53:33.895975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:33.895941 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9"] Oct 30 21:53:34.043639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.043608 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw"] Oct 30 21:53:34.044291 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044278 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ff7d2f5d-e789-46c6-a38a-f4c960126d85" containerName="revoke-keys" Oct 30 21:53:34.044349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044293 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7d2f5d-e789-46c6-a38a-f4c960126d85" containerName="revoke-keys" Oct 30 21:53:34.044349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044316 4526 cpu_manager.go:414] "containerMap: removing container" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" Oct 30 21:53:34.044349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044322 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" Oct 30 21:53:34.044349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044336 4526 cpu_manager.go:414] "containerMap: removing container" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" Oct 30 21:53:34.044349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044340 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" Oct 30 21:53:34.044554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044448 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7d2f5d-e789-46c6-a38a-f4c960126d85" containerName="revoke-keys" Oct 30 21:53:34.044554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044458 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" Oct 30 21:53:34.044554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.044468 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaba0bf1-59e8-4cd1-ab2f-173d71fdf7fb" containerName="istio-proxy" Oct 30 21:53:34.049837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.049818 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.107073 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.107033 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw"] Oct 30 21:53:34.108442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.107559 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:34.108442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.108222 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:53:34.108680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.108278 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:53:34.108680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.108352 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:53:34.237595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.237515 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.238671 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.238642 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.240455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.240286 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrjg\" (UniqueName: \"kubernetes.io/projected/e48764c5-8809-486d-90ab-d761bd31fcd1-kube-api-access-vxrjg\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.342115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.342022 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrjg\" (UniqueName: \"kubernetes.io/projected/e48764c5-8809-486d-90ab-d761bd31fcd1-kube-api-access-vxrjg\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.342315 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.342124 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.342315 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.342186 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.342987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.342959 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.344784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.344760 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.400441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.400390 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrjg\" (UniqueName: \"kubernetes.io/projected/e48764c5-8809-486d-90ab-d761bd31fcd1-kube-api-access-vxrjg\") pod \"maas-api-revoke-keys-e2e-api-a-b34958-c8vgw\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:34.664422 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:34.664303 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:35.073451 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:53:35.073416 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48764c5_8809_486d_90ab_d761bd31fcd1.slice/crio-34e8f790e4f5b0e3e05986350bfad18a2f0b861de0a5349f7d545a846a903eb8 WatchSource:0}: Error finding container 34e8f790e4f5b0e3e05986350bfad18a2f0b861de0a5349f7d545a846a903eb8: Status 404 returned error can't find the container with id 34e8f790e4f5b0e3e05986350bfad18a2f0b861de0a5349f7d545a846a903eb8 Oct 30 21:53:35.074823 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:35.074797 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw"] Oct 30 21:53:35.281438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:35.281399 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" event={"ID":"e48764c5-8809-486d-90ab-d761bd31fcd1","Type":"ContainerStarted","Data":"34e8f790e4f5b0e3e05986350bfad18a2f0b861de0a5349f7d545a846a903eb8"} Oct 30 21:53:36.291373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:36.291336 4526 generic.go:356] "Generic (PLEG): container finished" podID="e48764c5-8809-486d-90ab-d761bd31fcd1" containerID="2b8c5cc9c9792198c213061f311738945e3c2a6cfd596bc8bfa80c3f05810c3c" exitCode=0 Oct 30 21:53:36.291899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:36.291375 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" event={"ID":"e48764c5-8809-486d-90ab-d761bd31fcd1","Type":"ContainerDied","Data":"2b8c5cc9c9792198c213061f311738945e3c2a6cfd596bc8bfa80c3f05810c3c"} Oct 30 21:53:37.477694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.477669 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:37.513474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.513445 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48764c5-8809-486d-90ab-d761bd31fcd1-kube-api-access-vxrjg\" (UniqueName: \"kubernetes.io/projected/e48764c5-8809-486d-90ab-d761bd31fcd1-kube-api-access-vxrjg\") pod \"e48764c5-8809-486d-90ab-d761bd31fcd1\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " Oct 30 21:53:37.513668 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.513483 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-xks-ca\") pod \"e48764c5-8809-486d-90ab-d761bd31fcd1\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " Oct 30 21:53:37.513806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.513790 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-openshift-ca\") pod \"e48764c5-8809-486d-90ab-d761bd31fcd1\" (UID: \"e48764c5-8809-486d-90ab-d761bd31fcd1\") " Oct 30 21:53:37.514101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.514070 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-openshift-ca" pod "e48764c5-8809-486d-90ab-d761bd31fcd1" (UID: "e48764c5-8809-486d-90ab-d761bd31fcd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:37.514382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.514325 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:37.515824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.515799 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-xks-ca" pod "e48764c5-8809-486d-90ab-d761bd31fcd1" (UID: "e48764c5-8809-486d-90ab-d761bd31fcd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:37.515945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.515870 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48764c5-8809-486d-90ab-d761bd31fcd1-kube-api-access-vxrjg" pod "e48764c5-8809-486d-90ab-d761bd31fcd1" (UID: "e48764c5-8809-486d-90ab-d761bd31fcd1"). InnerVolumeSpecName "kube-api-access-vxrjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:37.616303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.616213 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxrjg\" (UniqueName: \"kubernetes.io/projected/e48764c5-8809-486d-90ab-d761bd31fcd1-kube-api-access-vxrjg\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:37.616303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:37.616244 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48764c5-8809-486d-90ab-d761bd31fcd1-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.309326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:38.309291 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" Oct 30 21:53:38.309541 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:38.309327 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw" event={"ID":"e48764c5-8809-486d-90ab-d761bd31fcd1","Type":"ContainerDied","Data":"34e8f790e4f5b0e3e05986350bfad18a2f0b861de0a5349f7d545a846a903eb8"} Oct 30 21:53:38.309541 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:38.309377 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e8f790e4f5b0e3e05986350bfad18a2f0b861de0a5349f7d545a846a903eb8" Oct 30 21:53:38.779681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:38.779561 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" podUID="2820f13c-a842-4d97-94ab-6cf0198c8cc1" containerName="payload-processing" containerID="cri-o://f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31" gracePeriod=25 Oct 30 21:53:38.896284 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:38.896241 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" podUID="e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df" containerName="payload-pre-processing" containerID="cri-o://e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a" gracePeriod=25 Oct 30 21:53:39.085520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.085429 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:53:39.139106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.139068 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2820f13c-a842-4d97-94ab-6cf0198c8cc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2820f13c-a842-4d97-94ab-6cf0198c8cc1-plugins-config-volume\") pod \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\" (UID: \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\") " Oct 30 21:53:39.139311 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.139238 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2820f13c-a842-4d97-94ab-6cf0198c8cc1-kube-api-access-hwxl7\" (UniqueName: \"kubernetes.io/projected/2820f13c-a842-4d97-94ab-6cf0198c8cc1-kube-api-access-hwxl7\") pod \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\" (UID: \"2820f13c-a842-4d97-94ab-6cf0198c8cc1\") " Oct 30 21:53:39.139443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.139421 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2820f13c-a842-4d97-94ab-6cf0198c8cc1-plugins-config-volume" pod "2820f13c-a842-4d97-94ab-6cf0198c8cc1" (UID: "2820f13c-a842-4d97-94ab-6cf0198c8cc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:39.140266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.140238 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2820f13c-a842-4d97-94ab-6cf0198c8cc1-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:39.142776 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.142751 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2820f13c-a842-4d97-94ab-6cf0198c8cc1-kube-api-access-hwxl7" pod "2820f13c-a842-4d97-94ab-6cf0198c8cc1" (UID: "2820f13c-a842-4d97-94ab-6cf0198c8cc1"). InnerVolumeSpecName "kube-api-access-hwxl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:39.180261 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.180238 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:53:39.241684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.241653 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-kube-api-access-scg24\" (UniqueName: \"kubernetes.io/projected/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-kube-api-access-scg24\") pod \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\" (UID: \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\") " Oct 30 21:53:39.241903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.241805 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-plugins-config-volume\") pod \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\" (UID: \"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df\") " Oct 30 21:53:39.242451 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.242373 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-plugins-config-volume" pod "e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df" (UID: "e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:39.242739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.242546 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwxl7\" (UniqueName: \"kubernetes.io/projected/2820f13c-a842-4d97-94ab-6cf0198c8cc1-kube-api-access-hwxl7\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:39.242739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.242561 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:39.244158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.244131 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-kube-api-access-scg24" pod "e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df" (UID: "e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df"). InnerVolumeSpecName "kube-api-access-scg24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:39.318699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.318601 4526 generic.go:356] "Generic (PLEG): container finished" podID="e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df" containerID="e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a" exitCode=0 Oct 30 21:53:39.318699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.318654 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" Oct 30 21:53:39.318699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.318686 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" event={"ID":"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df","Type":"ContainerDied","Data":"e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a"} Oct 30 21:53:39.318991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.318751 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9" event={"ID":"e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df","Type":"ContainerDied","Data":"2b5f16b681d3c47c4f68a79ccc0f6ed78a1c03854d6ad65090aaf0e878fba2fc"} Oct 30 21:53:39.318991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.318781 4526 scope.go:122] "RemoveContainer" containerID="e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a" Oct 30 21:53:39.320288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.320265 4526 generic.go:356] "Generic (PLEG): container finished" podID="2820f13c-a842-4d97-94ab-6cf0198c8cc1" containerID="f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31" exitCode=0 Oct 30 21:53:39.320410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.320322 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" event={"ID":"2820f13c-a842-4d97-94ab-6cf0198c8cc1","Type":"ContainerDied","Data":"f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31"} Oct 30 21:53:39.320410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.320364 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" event={"ID":"2820f13c-a842-4d97-94ab-6cf0198c8cc1","Type":"ContainerDied","Data":"371794bab5655c07c93da5c9efe51e9c21bb6c634693376170c8413865057b59"} Oct 30 21:53:39.320410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.320329 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl" Oct 30 21:53:39.333669 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.333632 4526 scope.go:122] "RemoveContainer" containerID="e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a" Oct 30 21:53:39.334005 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:53:39.333976 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a\": container with ID starting with e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a not found: ID does not exist" containerID="e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a" Oct 30 21:53:39.334117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.334014 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a"} err="failed to get container status \"e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a\": rpc error: code = NotFound desc = could not find container \"e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a\": container with ID starting with e0cd974fad76e3b721e6f96fedff2a5c85cb7594200a076fc4b9d4fa8e39339a not found: ID does not exist" Oct 30 21:53:39.334117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.334038 4526 scope.go:122] "RemoveContainer" containerID="f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31" Oct 30 21:53:39.344264 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.344234 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scg24\" (UniqueName: \"kubernetes.io/projected/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df-kube-api-access-scg24\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:39.348241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.348218 4526 scope.go:122] "RemoveContainer" containerID="f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31" Oct 30 21:53:39.348637 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:53:39.348604 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31\": container with ID starting with f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31 not found: ID does not exist" containerID="f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31" Oct 30 21:53:39.348716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.348647 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31"} err="failed to get container status \"f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31\": rpc error: code = NotFound desc = could not find container \"f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31\": container with ID starting with f509a8740f55d34acf3dea24fa5cc5df0219888eec6b6c4f65174074b3a5ce31 not found: ID does not exist" Oct 30 21:53:39.554613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.554566 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl"] Oct 30 21:53:39.558475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.558447 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-013fb2-5c54856855-8fvtl"] Oct 30 21:53:39.683327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.683239 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9"] Oct 30 21:53:39.692770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.692740 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-013fb2-78d76d7c99-lfxb9"] Oct 30 21:53:39.986911 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.986831 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2820f13c-a842-4d97-94ab-6cf0198c8cc1" path="/var/lib/kubelet/pods/2820f13c-a842-4d97-94ab-6cf0198c8cc1/volumes" Oct 30 21:53:39.987299 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:39.987285 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df" path="/var/lib/kubelet/pods/e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df/volumes" Oct 30 21:53:42.480763 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:53:42.480725 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:53:42.950770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:42.950734 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k"] Oct 30 21:53:43.000114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:43.000082 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b188-2rx8k"] Oct 30 21:53:43.987147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:43.987107 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b" path="/var/lib/kubelet/pods/e6bf16b7-fa6e-45a6-9ff6-dae6d1924c7b/volumes" Oct 30 21:53:45.555800 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:45.555770 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk"] Oct 30 21:53:45.685228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:45.685202 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp"] Oct 30 21:53:50.556345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.556303 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" podUID="83150ced-9ed7-40b2-8472-c9585e6258c8" containerName="payload-processing" containerID="cri-o://3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908" gracePeriod=25 Oct 30 21:53:50.685908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.685865 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" podUID="7106f4b4-33a0-40bb-b75a-614d8002c293" containerName="payload-pre-processing" containerID="cri-o://0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55" gracePeriod=25 Oct 30 21:53:50.880425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.880393 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:53:50.940847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.940808 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83150ced-9ed7-40b2-8472-c9585e6258c8-kube-api-access-g8xzv\" (UniqueName: \"kubernetes.io/projected/83150ced-9ed7-40b2-8472-c9585e6258c8-kube-api-access-g8xzv\") pod \"83150ced-9ed7-40b2-8472-c9585e6258c8\" (UID: \"83150ced-9ed7-40b2-8472-c9585e6258c8\") " Oct 30 21:53:50.941039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.940892 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83150ced-9ed7-40b2-8472-c9585e6258c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83150ced-9ed7-40b2-8472-c9585e6258c8-plugins-config-volume\") pod \"83150ced-9ed7-40b2-8472-c9585e6258c8\" (UID: \"83150ced-9ed7-40b2-8472-c9585e6258c8\") " Oct 30 21:53:50.941368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.941336 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83150ced-9ed7-40b2-8472-c9585e6258c8-plugins-config-volume" pod "83150ced-9ed7-40b2-8472-c9585e6258c8" (UID: "83150ced-9ed7-40b2-8472-c9585e6258c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:50.941750 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.941724 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83150ced-9ed7-40b2-8472-c9585e6258c8-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:50.944402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.944267 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83150ced-9ed7-40b2-8472-c9585e6258c8-kube-api-access-g8xzv" pod "83150ced-9ed7-40b2-8472-c9585e6258c8" (UID: "83150ced-9ed7-40b2-8472-c9585e6258c8"). InnerVolumeSpecName "kube-api-access-g8xzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:50.968802 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:50.968774 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:53:51.042727 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.042690 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7106f4b4-33a0-40bb-b75a-614d8002c293-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7106f4b4-33a0-40bb-b75a-614d8002c293-plugins-config-volume\") pod \"7106f4b4-33a0-40bb-b75a-614d8002c293\" (UID: \"7106f4b4-33a0-40bb-b75a-614d8002c293\") " Oct 30 21:53:51.042914 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.042874 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7106f4b4-33a0-40bb-b75a-614d8002c293-kube-api-access-c9r6d\" (UniqueName: \"kubernetes.io/projected/7106f4b4-33a0-40bb-b75a-614d8002c293-kube-api-access-c9r6d\") pod \"7106f4b4-33a0-40bb-b75a-614d8002c293\" (UID: \"7106f4b4-33a0-40bb-b75a-614d8002c293\") " Oct 30 21:53:51.043180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.043145 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7106f4b4-33a0-40bb-b75a-614d8002c293-plugins-config-volume" pod "7106f4b4-33a0-40bb-b75a-614d8002c293" (UID: "7106f4b4-33a0-40bb-b75a-614d8002c293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:51.044211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.044033 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8xzv\" (UniqueName: \"kubernetes.io/projected/83150ced-9ed7-40b2-8472-c9585e6258c8-kube-api-access-g8xzv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:51.044211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.044060 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7106f4b4-33a0-40bb-b75a-614d8002c293-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:51.045417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.045395 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7106f4b4-33a0-40bb-b75a-614d8002c293-kube-api-access-c9r6d" pod "7106f4b4-33a0-40bb-b75a-614d8002c293" (UID: "7106f4b4-33a0-40bb-b75a-614d8002c293"). InnerVolumeSpecName "kube-api-access-c9r6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:51.145626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.145536 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9r6d\" (UniqueName: \"kubernetes.io/projected/7106f4b4-33a0-40bb-b75a-614d8002c293-kube-api-access-c9r6d\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:51.463598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.463506 4526 generic.go:356] "Generic (PLEG): container finished" podID="7106f4b4-33a0-40bb-b75a-614d8002c293" containerID="0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55" exitCode=0 Oct 30 21:53:51.463598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.463550 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" Oct 30 21:53:51.463598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.463584 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" event={"ID":"7106f4b4-33a0-40bb-b75a-614d8002c293","Type":"ContainerDied","Data":"0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55"} Oct 30 21:53:51.463887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.463642 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp" event={"ID":"7106f4b4-33a0-40bb-b75a-614d8002c293","Type":"ContainerDied","Data":"fd17f6a0f44c53d67be8984f4305b7b32eeea248e53fbabd0cab6770c88cf4c8"} Oct 30 21:53:51.463887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.463666 4526 scope.go:122] "RemoveContainer" containerID="0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55" Oct 30 21:53:51.465094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.465075 4526 generic.go:356] "Generic (PLEG): container finished" podID="83150ced-9ed7-40b2-8472-c9585e6258c8" containerID="3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908" exitCode=0 Oct 30 21:53:51.465221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.465110 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" event={"ID":"83150ced-9ed7-40b2-8472-c9585e6258c8","Type":"ContainerDied","Data":"3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908"} Oct 30 21:53:51.465221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.465170 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" event={"ID":"83150ced-9ed7-40b2-8472-c9585e6258c8","Type":"ContainerDied","Data":"b343f81160d1a2b8bbf539c411e39601121cfaae824c2d6ccd873f2be0fd8627"} Oct 30 21:53:51.465221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.465122 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk" Oct 30 21:53:51.474619 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.474522 4526 scope.go:122] "RemoveContainer" containerID="0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55" Oct 30 21:53:51.474828 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:53:51.474806 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55\": container with ID starting with 0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55 not found: ID does not exist" containerID="0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55" Oct 30 21:53:51.474944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.474837 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55"} err="failed to get container status \"0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55\": rpc error: code = NotFound desc = could not find container \"0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55\": container with ID starting with 0a77d09966de9ac4c53b023794faef57e75727370e7369a5d7b29f65188d6b55 not found: ID does not exist" Oct 30 21:53:51.474944 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.474859 4526 scope.go:122] "RemoveContainer" containerID="3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908" Oct 30 21:53:51.484500 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.484469 4526 scope.go:122] "RemoveContainer" containerID="3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908" Oct 30 21:53:51.484807 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:53:51.484790 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908\": container with ID starting with 3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908 not found: ID does not exist" containerID="3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908" Oct 30 21:53:51.484854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.484817 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908"} err="failed to get container status \"3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908\": rpc error: code = NotFound desc = could not find container \"3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908\": container with ID starting with 3ebd71da9c350a72c02a45da8abe5cb3f75a1aeaf6f7b6d13a10d69d320ab908 not found: ID does not exist" Oct 30 21:53:51.638101 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.638067 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp"] Oct 30 21:53:51.643074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.643051 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b34958-74cf77fb79-rb9rp"] Oct 30 21:53:51.795629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.795597 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk"] Oct 30 21:53:51.807243 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.807216 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34958-9786989c7-lcprk"] Oct 30 21:53:51.807999 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.807981 4526 scope.go:122] "RemoveContainer" containerID="e93b17493fee417b4f6528178c4e53a43cde41cd294d1f86fa24e02d6735959a" Oct 30 21:53:51.818779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.818759 4526 scope.go:122] "RemoveContainer" containerID="c116c9bbaa7d93228c1dc0af9983fbef6e9684b4065deea5bd95cc060ff327b6" Oct 30 21:53:51.986562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.986530 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7106f4b4-33a0-40bb-b75a-614d8002c293" path="/var/lib/kubelet/pods/7106f4b4-33a0-40bb-b75a-614d8002c293/volumes" Oct 30 21:53:51.986979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:51.986966 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83150ced-9ed7-40b2-8472-c9585e6258c8" path="/var/lib/kubelet/pods/83150ced-9ed7-40b2-8472-c9585e6258c8/volumes" Oct 30 21:53:59.753038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:59.752939 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4"] Oct 30 21:53:59.802449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:59.802418 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d7f4c7-5tcd4"] Oct 30 21:53:59.987432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:53:59.987399 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b11d68-9460-470d-bae5-4528db915f54" path="/var/lib/kubelet/pods/d4b11d68-9460-470d-bae5-4528db915f54/volumes" Oct 30 21:54:00.634369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:00.634328 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8"] Oct 30 21:54:00.634677 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:00.634634 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" containerID="cri-o://58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e" gracePeriod=30 Oct 30 21:54:00.874280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:00.874247 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f"] Oct 30 21:54:00.874762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:00.874479 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" containerID="cri-o://2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60" gracePeriod=30 Oct 30 21:54:02.250393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:02.250351 4526 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:02.250821 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:02.250417 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:02.583143 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:02.583107 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:02.583312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:02.583170 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:03.660920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.660881 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld"] Oct 30 21:54:03.662455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.662427 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df" containerName="payload-pre-processing" Oct 30 21:54:03.662670 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.662654 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df" containerName="payload-pre-processing" Oct 30 21:54:03.662770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.662759 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e48764c5-8809-486d-90ab-d761bd31fcd1" containerName="revoke-keys" Oct 30 21:54:03.662839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.662829 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48764c5-8809-486d-90ab-d761bd31fcd1" containerName="revoke-keys" Oct 30 21:54:03.662933 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.662921 4526 cpu_manager.go:414] "containerMap: removing container" podUID="2820f13c-a842-4d97-94ab-6cf0198c8cc1" containerName="payload-processing" Oct 30 21:54:03.663007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.662998 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2820f13c-a842-4d97-94ab-6cf0198c8cc1" containerName="payload-processing" Oct 30 21:54:03.663093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663082 4526 cpu_manager.go:414] "containerMap: removing container" podUID="83150ced-9ed7-40b2-8472-c9585e6258c8" containerName="payload-processing" Oct 30 21:54:03.663172 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663163 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83150ced-9ed7-40b2-8472-c9585e6258c8" containerName="payload-processing" Oct 30 21:54:03.663267 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663258 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7106f4b4-33a0-40bb-b75a-614d8002c293" containerName="payload-pre-processing" Oct 30 21:54:03.663341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663331 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7106f4b4-33a0-40bb-b75a-614d8002c293" containerName="payload-pre-processing" Oct 30 21:54:03.663694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663675 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7106f4b4-33a0-40bb-b75a-614d8002c293" containerName="payload-pre-processing" Oct 30 21:54:03.663888 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663874 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3aa1bb4-7e94-4af0-b9ec-e1c24729a7df" containerName="payload-pre-processing" Oct 30 21:54:03.663979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663960 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48764c5-8809-486d-90ab-d761bd31fcd1" containerName="revoke-keys" Oct 30 21:54:03.663979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663981 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="2820f13c-a842-4d97-94ab-6cf0198c8cc1" containerName="payload-processing" Oct 30 21:54:03.664179 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.663996 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="83150ced-9ed7-40b2-8472-c9585e6258c8" containerName="payload-processing" Oct 30 21:54:03.673252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.673222 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:03.726313 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.726272 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld"] Oct 30 21:54:03.727001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.726849 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:54:03.766535 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.766503 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:03.766725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.766601 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:03.766725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.766648 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:03.857708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.857670 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5s8z\" (UniqueName: \"kubernetes.io/projected/078567c1-6c75-4bf7-91f4-046a7c3450bc-kube-api-access-n5s8z\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:03.858087 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.858063 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:03.858283 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.858263 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:03.969560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.964567 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5s8z\" (UniqueName: \"kubernetes.io/projected/078567c1-6c75-4bf7-91f4-046a7c3450bc-kube-api-access-n5s8z\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:03.969560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.964683 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:03.969560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.964756 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:03.969560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.968291 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:03.971075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:03.971046 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:04.028560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:04.028455 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5s8z\" (UniqueName: \"kubernetes.io/projected/078567c1-6c75-4bf7-91f4-046a7c3450bc-kube-api-access-n5s8z\") pod \"maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:04.300177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:04.300146 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:04.691474 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:04.691441 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld"] Oct 30 21:54:04.691908 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:54:04.691831 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078567c1_6c75_4bf7_91f4_046a7c3450bc.slice/crio-ec007afffe6be6286c61f3f197de5041f322d214fd9df2c4187a380695c0c30b WatchSource:0}: Error finding container ec007afffe6be6286c61f3f197de5041f322d214fd9df2c4187a380695c0c30b: Status 404 returned error can't find the container with id ec007afffe6be6286c61f3f197de5041f322d214fd9df2c4187a380695c0c30b Oct 30 21:54:05.591547 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:05.591480 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" event={"ID":"078567c1-6c75-4bf7-91f4-046a7c3450bc","Type":"ContainerStarted","Data":"86905f6d50fabf32da05acb7fd45e1b388c2c10c0d4ee18e934c3a52293708ea"} Oct 30 21:54:05.591743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:05.591566 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" event={"ID":"078567c1-6c75-4bf7-91f4-046a7c3450bc","Type":"ContainerStarted","Data":"ec007afffe6be6286c61f3f197de5041f322d214fd9df2c4187a380695c0c30b"} Oct 30 21:54:05.713741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:05.713670 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" podStartSLOduration=2.713649727 podStartE2EDuration="2.713649727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:05.708448293 +0000 UTC m=+2235.841821493" watchObservedRunningTime="2026-10-30 21:54:05.713649727 +0000 UTC m=+2235.847022926" Oct 30 21:54:05.737746 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:05.737717 4526 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078567c1_6c75_4bf7_91f4_046a7c3450bc.slice/crio-86905f6d50fabf32da05acb7fd45e1b388c2c10c0d4ee18e934c3a52293708ea.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:54:05.737847 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:05.737743 4526 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078567c1_6c75_4bf7_91f4_046a7c3450bc.slice/crio-conmon-86905f6d50fabf32da05acb7fd45e1b388c2c10c0d4ee18e934c3a52293708ea.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:54:06.000210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.000180 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:54:06.096067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.096031 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-data\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.096296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.096107 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-istio-token\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-istio-token\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.096296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.096236 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac25a73-df68-4c82-989d-2dd28e379484-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac25a73-df68-4c82-989d-2dd28e379484-istiod-ca-cert\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.096296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.096284 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-socket\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.096639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.096319 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-kube-api-access-9g6kv\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-kube-api-access-9g6kv\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.096639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.096376 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-envoy\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.096639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.096504 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-certs\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.096639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.096562 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dac25a73-df68-4c82-989d-2dd28e379484-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac25a73-df68-4c82-989d-2dd28e379484-istio-podinfo\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.097149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.097120 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac25a73-df68-4c82-989d-2dd28e379484-istiod-ca-cert" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:06.097377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.097359 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-socket" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.097461 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.097408 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-credential-socket\") pod \"dac25a73-df68-4c82-989d-2dd28e379484\" (UID: \"dac25a73-df68-4c82-989d-2dd28e379484\") " Oct 30 21:54:06.098248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.097992 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dac25a73-df68-4c82-989d-2dd28e379484-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.098248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.098016 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.098405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.098285 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-credential-socket" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.098405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.098331 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-data" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.098615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.098591 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-certs" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.099772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.099569 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-kube-api-access-9g6kv" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "kube-api-access-9g6kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:06.100121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.100103 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-istio-token" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:06.100349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.100325 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dac25a73-df68-4c82-989d-2dd28e379484-istio-podinfo" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:06.102552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.102528 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-envoy" pod "dac25a73-df68-4c82-989d-2dd28e379484" (UID: "dac25a73-df68-4c82-989d-2dd28e379484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.192903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.192877 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:54:06.199204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.199177 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.199204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.199205 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dac25a73-df68-4c82-989d-2dd28e379484-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.199390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.199215 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.199390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.199226 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.199390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.199235 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.199390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.199243 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6kv\" (UniqueName: \"kubernetes.io/projected/dac25a73-df68-4c82-989d-2dd28e379484-kube-api-access-9g6kv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.199390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.199252 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dac25a73-df68-4c82-989d-2dd28e379484-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.300270 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.300240 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-data\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.300457 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.300311 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-certs\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.300457 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.300392 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istiod-ca-cert\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.300640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.300587 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-podinfo\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.300700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.300681 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-certs" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.300762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.300712 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-data" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.301071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.301032 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istiod-ca-cert" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:06.301221 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.301194 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-kube-api-access-z8rtv\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-kube-api-access-z8rtv\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.301311 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.301257 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-envoy\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.301378 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.301317 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-credential-socket\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.301473 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-socket\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.301586 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-token\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-token\") pod \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\" (UID: \"470b0a52-6093-4cb9-85ef-ce5b3bd93485\") " Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.301715 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-credential-socket" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.301907 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-socket" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.302310 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.302331 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.302347 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.302361 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.303162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.302375 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.306481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.306437 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-envoy" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:06.306680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.306453 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-podinfo" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:06.306680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.306633 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-kube-api-access-z8rtv" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "kube-api-access-z8rtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:06.307193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.307169 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-token" pod "470b0a52-6093-4cb9-85ef-ce5b3bd93485" (UID: "470b0a52-6093-4cb9-85ef-ce5b3bd93485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:06.404732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.404621 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8rtv\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-kube-api-access-z8rtv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.404732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.404670 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.404732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.404685 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.404732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.404698 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/470b0a52-6093-4cb9-85ef-ce5b3bd93485-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:06.599522 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.599468 4526 generic.go:356] "Generic (PLEG): container finished" podID="dac25a73-df68-4c82-989d-2dd28e379484" containerID="58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e" exitCode=0 Oct 30 21:54:06.599731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.599567 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" Oct 30 21:54:06.599731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.599563 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" event={"ID":"dac25a73-df68-4c82-989d-2dd28e379484","Type":"ContainerDied","Data":"58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e"} Oct 30 21:54:06.599731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.599629 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8" event={"ID":"dac25a73-df68-4c82-989d-2dd28e379484","Type":"ContainerDied","Data":"16ab9dae772c8d72271cc3cdf54293fbac8ac3438e37596588ed8d07877ef21c"} Oct 30 21:54:06.599731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.599658 4526 scope.go:122] "RemoveContainer" containerID="58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e" Oct 30 21:54:06.601518 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.601474 4526 generic.go:356] "Generic (PLEG): container finished" podID="078567c1-6c75-4bf7-91f4-046a7c3450bc" containerID="86905f6d50fabf32da05acb7fd45e1b388c2c10c0d4ee18e934c3a52293708ea" exitCode=0 Oct 30 21:54:06.601643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.601562 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" event={"ID":"078567c1-6c75-4bf7-91f4-046a7c3450bc","Type":"ContainerDied","Data":"86905f6d50fabf32da05acb7fd45e1b388c2c10c0d4ee18e934c3a52293708ea"} Oct 30 21:54:06.603070 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.603046 4526 generic.go:356] "Generic (PLEG): container finished" podID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerID="2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60" exitCode=0 Oct 30 21:54:06.603192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.603091 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" Oct 30 21:54:06.603192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.603125 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" event={"ID":"470b0a52-6093-4cb9-85ef-ce5b3bd93485","Type":"ContainerDied","Data":"2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60"} Oct 30 21:54:06.603192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.603166 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f" event={"ID":"470b0a52-6093-4cb9-85ef-ce5b3bd93485","Type":"ContainerDied","Data":"77cd2b15524790149d2c4a7bd97d8d90877ded116bc590b30ce97ad7cf61939b"} Oct 30 21:54:06.611988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.611968 4526 scope.go:122] "RemoveContainer" containerID="58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e" Oct 30 21:54:06.612293 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:06.612248 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e\": container with ID starting with 58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e not found: ID does not exist" containerID="58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e" Oct 30 21:54:06.612379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.612284 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e"} err="failed to get container status \"58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e\": rpc error: code = NotFound desc = could not find container \"58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e\": container with ID starting with 58e32c9ab336884a8e6e8397941702b91278407faa5aa63f4d09fde7db2b5b7e not found: ID does not exist" Oct 30 21:54:06.612379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.612309 4526 scope.go:122] "RemoveContainer" containerID="2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60" Oct 30 21:54:06.621831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.621813 4526 scope.go:122] "RemoveContainer" containerID="2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60" Oct 30 21:54:06.622121 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:06.622102 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60\": container with ID starting with 2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60 not found: ID does not exist" containerID="2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60" Oct 30 21:54:06.622210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.622129 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60"} err="failed to get container status \"2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60\": rpc error: code = NotFound desc = could not find container \"2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60\": container with ID starting with 2cce4b7f460aeb0e687cf01d11206a31716efde5513b6c401339e9800f76ce60 not found: ID does not exist" Oct 30 21:54:06.766700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.766615 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f"] Oct 30 21:54:06.815749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.815719 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-j6j7f"] Oct 30 21:54:06.879355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.879321 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8"] Oct 30 21:54:06.882893 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:06.882871 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-4ljh8"] Oct 30 21:54:07.034408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.034383 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj"] Oct 30 21:54:07.049964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.049929 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw"] Oct 30 21:54:07.776147 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.776117 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:07.818410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.815995 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2"] Oct 30 21:54:07.818410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.816252 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" containerID="cri-o://33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71" gracePeriod=30 Oct 30 21:54:07.835199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.835166 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-openshift-ca\") pod \"078567c1-6c75-4bf7-91f4-046a7c3450bc\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " Oct 30 21:54:07.846817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.835362 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078567c1-6c75-4bf7-91f4-046a7c3450bc-kube-api-access-n5s8z\" (UniqueName: \"kubernetes.io/projected/078567c1-6c75-4bf7-91f4-046a7c3450bc-kube-api-access-n5s8z\") pod \"078567c1-6c75-4bf7-91f4-046a7c3450bc\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " Oct 30 21:54:07.846817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.835432 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-xks-ca\") pod \"078567c1-6c75-4bf7-91f4-046a7c3450bc\" (UID: \"078567c1-6c75-4bf7-91f4-046a7c3450bc\") " Oct 30 21:54:07.846817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.835855 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-openshift-ca" pod "078567c1-6c75-4bf7-91f4-046a7c3450bc" (UID: "078567c1-6c75-4bf7-91f4-046a7c3450bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:07.846817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.836094 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:07.846817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.841591 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-xks-ca" pod "078567c1-6c75-4bf7-91f4-046a7c3450bc" (UID: "078567c1-6c75-4bf7-91f4-046a7c3450bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:07.846817 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.841667 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078567c1-6c75-4bf7-91f4-046a7c3450bc-kube-api-access-n5s8z" pod "078567c1-6c75-4bf7-91f4-046a7c3450bc" (UID: "078567c1-6c75-4bf7-91f4-046a7c3450bc"). InnerVolumeSpecName "kube-api-access-n5s8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:07.937452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.937311 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5s8z\" (UniqueName: \"kubernetes.io/projected/078567c1-6c75-4bf7-91f4-046a7c3450bc-kube-api-access-n5s8z\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:07.937452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.937359 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078567c1-6c75-4bf7-91f4-046a7c3450bc-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:07.994591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.994454 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" path="/var/lib/kubelet/pods/470b0a52-6093-4cb9-85ef-ce5b3bd93485/volumes" Oct 30 21:54:07.995778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:07.995754 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac25a73-df68-4c82-989d-2dd28e379484" path="/var/lib/kubelet/pods/dac25a73-df68-4c82-989d-2dd28e379484/volumes" Oct 30 21:54:08.624027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:08.623995 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" Oct 30 21:54:08.624027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:08.624003 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld" event={"ID":"078567c1-6c75-4bf7-91f4-046a7c3450bc","Type":"ContainerDied","Data":"ec007afffe6be6286c61f3f197de5041f322d214fd9df2c4187a380695c0c30b"} Oct 30 21:54:08.624298 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:08.624087 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec007afffe6be6286c61f3f197de5041f322d214fd9df2c4187a380695c0c30b" Oct 30 21:54:11.693420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.693385 4526 patch_prober.go:29] interesting pod/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:11.693836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.693435 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:11.965370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.965296 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc"] Oct 30 21:54:11.965954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.965940 4526 cpu_manager.go:414] "containerMap: removing container" podUID="078567c1-6c75-4bf7-91f4-046a7c3450bc" containerName="revoke-keys" Oct 30 21:54:11.965995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.965956 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078567c1-6c75-4bf7-91f4-046a7c3450bc" containerName="revoke-keys" Oct 30 21:54:11.965995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.965967 4526 cpu_manager.go:414] "containerMap: removing container" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" Oct 30 21:54:11.965995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.965972 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" Oct 30 21:54:11.966098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.966011 4526 cpu_manager.go:414] "containerMap: removing container" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" Oct 30 21:54:11.966098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.966016 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" Oct 30 21:54:11.966158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.966114 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac25a73-df68-4c82-989d-2dd28e379484" containerName="istio-proxy" Oct 30 21:54:11.966158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.966126 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="078567c1-6c75-4bf7-91f4-046a7c3450bc" containerName="revoke-keys" Oct 30 21:54:11.966158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.966132 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="470b0a52-6093-4cb9-85ef-ce5b3bd93485" containerName="istio-proxy" Oct 30 21:54:11.970921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.970901 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:11.999844 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.999820 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:11.999983 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:11.999855 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8twn\" (UniqueName: \"kubernetes.io/projected/fe093fef-3a8f-4d8e-ab39-9eee95030bed-kube-api-access-c8twn\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.000103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.000082 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.029176 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.029144 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc"] Oct 30 21:54:12.074187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.074145 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:12.074187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.074182 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:12.075255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.075235 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:12.075358 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.075263 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:54:12.101417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.101389 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.101548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.101474 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.101548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.101525 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8twn\" (UniqueName: \"kubernetes.io/projected/fe093fef-3a8f-4d8e-ab39-9eee95030bed-kube-api-access-c8twn\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.102395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.102290 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.104064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.104037 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.158532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.158479 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8twn\" (UniqueName: \"kubernetes.io/projected/fe093fef-3a8f-4d8e-ab39-9eee95030bed-kube-api-access-c8twn\") pod \"maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.293744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.293718 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:12.480346 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:12.480305 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:54:12.685861 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:12.685835 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc"] Oct 30 21:54:12.687839 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:54:12.687811 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe093fef_3a8f_4d8e_ab39_9eee95030bed.slice/crio-34fffc2a88a7ab2ceec8f6e82d6e040504d3ecf573794ea9aa0ba03348a1b8fb WatchSource:0}: Error finding container 34fffc2a88a7ab2ceec8f6e82d6e040504d3ecf573794ea9aa0ba03348a1b8fb: Status 404 returned error can't find the container with id 34fffc2a88a7ab2ceec8f6e82d6e040504d3ecf573794ea9aa0ba03348a1b8fb Oct 30 21:54:13.513479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.513453 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:54:13.636440 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636411 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-podinfo\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.636632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636446 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-token\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-token\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.636632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636597 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-certs\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.636761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636651 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istiod-ca-cert\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.636761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636683 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-kube-api-access-5vbcq\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-kube-api-access-5vbcq\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.636761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636720 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-credential-socket\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.636928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636790 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-envoy\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.636928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636821 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-socket\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.636928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636860 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-data\") pod \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\" (UID: \"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec\") " Oct 30 21:54:13.637095 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.636930 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-certs" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:13.637355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637254 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-credential-socket" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:13.637504 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637415 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-socket" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:13.637580 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637537 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-data" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:13.637639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637614 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istiod-ca-cert" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:13.637772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637752 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.637838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637779 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.637838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637797 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.638013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637822 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.638013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.637932 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.639148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.639117 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-token" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:13.639262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.639209 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-podinfo" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:13.639262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.639231 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-kube-api-access-5vbcq" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "kube-api-access-5vbcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:13.639376 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.639357 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-envoy" pod "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" (UID: "e6093f34-95cd-453d-bc0e-8f7bc5ecfcec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:13.662651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.662627 4526 generic.go:356] "Generic (PLEG): container finished" podID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerID="33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71" exitCode=0 Oct 30 21:54:13.662743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.662680 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" Oct 30 21:54:13.662743 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.662689 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" event={"ID":"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec","Type":"ContainerDied","Data":"33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71"} Oct 30 21:54:13.662841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.662745 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2" event={"ID":"e6093f34-95cd-453d-bc0e-8f7bc5ecfcec","Type":"ContainerDied","Data":"c85851017e6723382c6f8760aba71b77b0ba8110bf40fb2d193c91d2589610aa"} Oct 30 21:54:13.662841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.662769 4526 scope.go:122] "RemoveContainer" containerID="33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71" Oct 30 21:54:13.664809 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.664784 4526 generic.go:356] "Generic (PLEG): container finished" podID="fe093fef-3a8f-4d8e-ab39-9eee95030bed" containerID="a9c578db721dee8796b91b61deb5ee43c632b004fa01359d19489b4cc4b8f998" exitCode=0 Oct 30 21:54:13.664931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.664817 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" event={"ID":"fe093fef-3a8f-4d8e-ab39-9eee95030bed","Type":"ContainerDied","Data":"a9c578db721dee8796b91b61deb5ee43c632b004fa01359d19489b4cc4b8f998"} Oct 30 21:54:13.664931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.664854 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" event={"ID":"fe093fef-3a8f-4d8e-ab39-9eee95030bed","Type":"ContainerStarted","Data":"34fffc2a88a7ab2ceec8f6e82d6e040504d3ecf573794ea9aa0ba03348a1b8fb"} Oct 30 21:54:13.692536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.692514 4526 scope.go:122] "RemoveContainer" containerID="a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13" Oct 30 21:54:13.704043 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.704017 4526 scope.go:122] "RemoveContainer" containerID="33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71" Oct 30 21:54:13.704380 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:13.704356 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71\": container with ID starting with 33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71 not found: ID does not exist" containerID="33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71" Oct 30 21:54:13.704503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.704387 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71"} err="failed to get container status \"33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71\": rpc error: code = NotFound desc = could not find container \"33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71\": container with ID starting with 33219fabfcfce44dd03242c25b9249cbfb351f9262d8b226e87a72324de83e71 not found: ID does not exist" Oct 30 21:54:13.704503 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.704406 4526 scope.go:122] "RemoveContainer" containerID="a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13" Oct 30 21:54:13.704722 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:13.704698 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13\": container with ID starting with a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13 not found: ID does not exist" containerID="a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13" Oct 30 21:54:13.704788 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.704731 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13"} err="failed to get container status \"a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13\": rpc error: code = NotFound desc = could not find container \"a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13\": container with ID starting with a0b86c5fd98151f712cf573ef1474def430f0cf6220b2b7b0765c4fdc26f9f13 not found: ID does not exist" Oct 30 21:54:13.739244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.739217 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.739244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.739245 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.739430 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.739255 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vbcq\" (UniqueName: \"kubernetes.io/projected/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-kube-api-access-5vbcq\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.739430 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.739265 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:13.948598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.948510 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2"] Oct 30 21:54:13.954473 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.954441 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d7f4c7-openshift-default-7754f4dcdf-h76j2"] Oct 30 21:54:13.986949 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:13.986908 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" path="/var/lib/kubelet/pods/e6093f34-95cd-453d-bc0e-8f7bc5ecfcec/volumes" Oct 30 21:54:14.848148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:14.848125 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:14.960213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:14.960125 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe093fef-3a8f-4d8e-ab39-9eee95030bed-kube-api-access-c8twn\" (UniqueName: \"kubernetes.io/projected/fe093fef-3a8f-4d8e-ab39-9eee95030bed-kube-api-access-c8twn\") pod \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " Oct 30 21:54:14.960395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:14.960240 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-openshift-ca\") pod \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " Oct 30 21:54:14.960395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:14.960296 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-xks-ca\") pod \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\" (UID: \"fe093fef-3a8f-4d8e-ab39-9eee95030bed\") " Oct 30 21:54:14.960687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:14.960655 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-openshift-ca" pod "fe093fef-3a8f-4d8e-ab39-9eee95030bed" (UID: "fe093fef-3a8f-4d8e-ab39-9eee95030bed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:14.960843 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:14.960830 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:14.962528 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:14.962477 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-xks-ca" pod "fe093fef-3a8f-4d8e-ab39-9eee95030bed" (UID: "fe093fef-3a8f-4d8e-ab39-9eee95030bed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:14.962632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:14.962596 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe093fef-3a8f-4d8e-ab39-9eee95030bed-kube-api-access-c8twn" pod "fe093fef-3a8f-4d8e-ab39-9eee95030bed" (UID: "fe093fef-3a8f-4d8e-ab39-9eee95030bed"). InnerVolumeSpecName "kube-api-access-c8twn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:15.062177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.062140 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe093fef-3a8f-4d8e-ab39-9eee95030bed-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.062177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.062172 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8twn\" (UniqueName: \"kubernetes.io/projected/fe093fef-3a8f-4d8e-ab39-9eee95030bed-kube-api-access-c8twn\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:15.155648 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.155604 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27"] Oct 30 21:54:15.155896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.155855 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" containerID="cri-o://f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d" gracePeriod=30 Oct 30 21:54:15.181862 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.181825 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt"] Oct 30 21:54:15.273555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.273436 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd"] Oct 30 21:54:15.682995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.682964 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" Oct 30 21:54:15.682995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.682979 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc" event={"ID":"fe093fef-3a8f-4d8e-ab39-9eee95030bed","Type":"ContainerDied","Data":"34fffc2a88a7ab2ceec8f6e82d6e040504d3ecf573794ea9aa0ba03348a1b8fb"} Oct 30 21:54:15.683196 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:15.683043 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34fffc2a88a7ab2ceec8f6e82d6e040504d3ecf573794ea9aa0ba03348a1b8fb" Oct 30 21:54:17.475244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:17.475211 4526 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:17.475890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:17.475272 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:20.030353 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.030314 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996"] Oct 30 21:54:20.030988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.030951 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" Oct 30 21:54:20.030988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.030965 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" Oct 30 21:54:20.031066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.031008 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fe093fef-3a8f-4d8e-ab39-9eee95030bed" containerName="revoke-keys" Oct 30 21:54:20.031066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.031014 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe093fef-3a8f-4d8e-ab39-9eee95030bed" containerName="revoke-keys" Oct 30 21:54:20.031137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.031099 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" Oct 30 21:54:20.031137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.031113 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" Oct 30 21:54:20.031137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.031120 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe093fef-3a8f-4d8e-ab39-9eee95030bed" containerName="revoke-keys" Oct 30 21:54:20.034931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.034912 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.093354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.093323 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:20.136337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.136294 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.136581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.136564 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8b44\" (UniqueName: \"kubernetes.io/projected/19533110-eb72-49f4-9dc6-7d65d282151a-kube-api-access-z8b44\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.136683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.136628 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.137849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.137830 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:54:20.137951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.137897 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:20.137951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.137917 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:20.142041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.142018 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996"] Oct 30 21:54:20.182429 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.182376 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" podUID="eb8f479d-3076-4853-b172-8a11f4721edd" containerName="payload-processing" containerID="cri-o://da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846" gracePeriod=25 Oct 30 21:54:20.191739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.191710 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm"] Oct 30 21:54:20.191976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.191945 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" podUID="52eda2ad-e32a-46e1-ae61-85d9a810f17c" containerName="discovery" containerID="cri-o://585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061" gracePeriod=30 Oct 30 21:54:20.238672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.238635 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8b44\" (UniqueName: \"kubernetes.io/projected/19533110-eb72-49f4-9dc6-7d65d282151a-kube-api-access-z8b44\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.238804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.238697 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.238848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.238818 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.239451 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.239433 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.241279 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.241258 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.274229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.274147 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" podUID="9611497d-715d-438f-98c1-d6b4332ef005" containerName="payload-pre-processing" containerID="cri-o://26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d" gracePeriod=25 Oct 30 21:54:20.304791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.304763 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8b44\" (UniqueName: \"kubernetes.io/projected/19533110-eb72-49f4-9dc6-7d65d282151a-kube-api-access-z8b44\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.349254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.349222 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:20.474107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.473968 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:54:20.529194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.529169 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:54:20.646327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646232 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-credential-socket\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.646327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646293 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-kube-api-access-b7g6c\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-kube-api-access-b7g6c\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.646606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646333 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-kubeconfig\") pod \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " Oct 30 21:54:20.646606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646353 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-data\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.646606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646383 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-kube-api-access-dm4ql\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-kube-api-access-dm4ql\") pod \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " Oct 30 21:54:20.646606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646418 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-ca-configmap\") pod \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " Oct 30 21:54:20.646606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646529 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-certs\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.646606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646555 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52eda2ad-e32a-46e1-ae61-85d9a810f17c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eda2ad-e32a-46e1-ae61-85d9a810f17c-local-certs\") pod \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " Oct 30 21:54:20.646606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646585 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-podinfo\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646616 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-dns-cert\") pod \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646627 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-credential-socket" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646731 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-token\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-token\") pod \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646791 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istiod-ca-cert\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646842 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-socket\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646857 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-data" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646958 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-cacerts\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-cacerts\") pod \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\" (UID: \"52eda2ad-e32a-46e1-ae61-85d9a810f17c\") " Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646969 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-ca-configmap" pod "52eda2ad-e32a-46e1-ae61-85d9a810f17c" (UID: "52eda2ad-e32a-46e1-ae61-85d9a810f17c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.646992 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-envoy\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.647048 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-token\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-token\") pod \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\" (UID: \"e8f64983-5fc2-4bcf-a97c-71de7483afa5\") " Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.647081 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-certs" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:20.647349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.647237 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-socket" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:20.649354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.647542 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istiod-ca-cert" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:20.649354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.647970 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.649354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.647992 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.649354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.648009 4526 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-ca-configmap\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.649354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.648027 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.649354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.648042 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.649354 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.648057 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.649816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.649667 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-kubeconfig" pod "52eda2ad-e32a-46e1-ae61-85d9a810f17c" (UID: "52eda2ad-e32a-46e1-ae61-85d9a810f17c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:20.649816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.649753 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-kube-api-access-dm4ql" pod "52eda2ad-e32a-46e1-ae61-85d9a810f17c" (UID: "52eda2ad-e32a-46e1-ae61-85d9a810f17c"). InnerVolumeSpecName "kube-api-access-dm4ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:20.652130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.652093 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-cacerts" pod "52eda2ad-e32a-46e1-ae61-85d9a810f17c" (UID: "52eda2ad-e32a-46e1-ae61-85d9a810f17c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:20.652481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.652334 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-dns-cert" pod "52eda2ad-e32a-46e1-ae61-85d9a810f17c" (UID: "52eda2ad-e32a-46e1-ae61-85d9a810f17c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:20.652481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.652385 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-kube-api-access-b7g6c" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "kube-api-access-b7g6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:20.652481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.652403 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52eda2ad-e32a-46e1-ae61-85d9a810f17c-local-certs" pod "52eda2ad-e32a-46e1-ae61-85d9a810f17c" (UID: "52eda2ad-e32a-46e1-ae61-85d9a810f17c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:20.652481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.652443 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-podinfo" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:20.652815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.652545 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-token" pod "52eda2ad-e32a-46e1-ae61-85d9a810f17c" (UID: "52eda2ad-e32a-46e1-ae61-85d9a810f17c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:20.652815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.652586 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-token" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:20.652815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.652690 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-envoy" pod "e8f64983-5fc2-4bcf-a97c-71de7483afa5" (UID: "e8f64983-5fc2-4bcf-a97c-71de7483afa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:20.730364 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.730324 4526 generic.go:356] "Generic (PLEG): container finished" podID="52eda2ad-e32a-46e1-ae61-85d9a810f17c" containerID="585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061" exitCode=0 Oct 30 21:54:20.731567 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.731537 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" Oct 30 21:54:20.731827 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.731625 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" event={"ID":"52eda2ad-e32a-46e1-ae61-85d9a810f17c","Type":"ContainerDied","Data":"585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061"} Oct 30 21:54:20.731827 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.731686 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm" event={"ID":"52eda2ad-e32a-46e1-ae61-85d9a810f17c","Type":"ContainerDied","Data":"4eb8a4f14613f9708549c8861f836291f9a73f120c18dad69839e997c957f43e"} Oct 30 21:54:20.731827 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.731730 4526 scope.go:122] "RemoveContainer" containerID="585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061" Oct 30 21:54:20.733377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.733266 4526 generic.go:356] "Generic (PLEG): container finished" podID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerID="f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d" exitCode=0 Oct 30 21:54:20.733377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.733333 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" Oct 30 21:54:20.733377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.733355 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" event={"ID":"e8f64983-5fc2-4bcf-a97c-71de7483afa5","Type":"ContainerDied","Data":"f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d"} Oct 30 21:54:20.733519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.733411 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27" event={"ID":"e8f64983-5fc2-4bcf-a97c-71de7483afa5","Type":"ContainerDied","Data":"cd64c799199ef473f71537d8f21a070d96bfa1da9e5070dac5f70fe08030bbce"} Oct 30 21:54:20.746379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.746321 4526 scope.go:122] "RemoveContainer" containerID="585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061" Oct 30 21:54:20.746736 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:20.746710 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061\": container with ID starting with 585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061 not found: ID does not exist" containerID="585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061" Oct 30 21:54:20.746818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.746747 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061"} err="failed to get container status \"585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061\": rpc error: code = NotFound desc = could not find container \"585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061\": container with ID starting with 585086fb1da176b26b7dfd221626ed1a3f891acce765b9dddd27ae92cac9a061 not found: ID does not exist" Oct 30 21:54:20.746818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.746768 4526 scope.go:122] "RemoveContainer" containerID="f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749548 4526 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-kubeconfig\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749573 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm4ql\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-kube-api-access-dm4ql\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749591 4526 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52eda2ad-e32a-46e1-ae61-85d9a810f17c-local-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749606 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749619 4526 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-csr-dns-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749630 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52eda2ad-e32a-46e1-ae61-85d9a810f17c-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749639 4526 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52eda2ad-e32a-46e1-ae61-85d9a810f17c-cacerts\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749648 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749667 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.749704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.749676 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7g6c\" (UniqueName: \"kubernetes.io/projected/e8f64983-5fc2-4bcf-a97c-71de7483afa5-kube-api-access-b7g6c\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:20.759150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.759122 4526 scope.go:122] "RemoveContainer" containerID="f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d" Oct 30 21:54:20.759522 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:20.759450 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d\": container with ID starting with f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d not found: ID does not exist" containerID="f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d" Oct 30 21:54:20.759629 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.759531 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d"} err="failed to get container status \"f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d\": rpc error: code = NotFound desc = could not find container \"f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d\": container with ID starting with f339fa509f21ea56d1bc608a1e1e04597079499c9b11438f95d9c196dfe8535d not found: ID does not exist" Oct 30 21:54:20.774763 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:54:20.774728 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19533110_eb72_49f4_9dc6_7d65d282151a.slice/crio-20d6c1d82158daa081434a3f5adc87a7671241ff56cd810e3733f5471f6ea5d6 WatchSource:0}: Error finding container 20d6c1d82158daa081434a3f5adc87a7671241ff56cd810e3733f5471f6ea5d6: Status 404 returned error can't find the container with id 20d6c1d82158daa081434a3f5adc87a7671241ff56cd810e3733f5471f6ea5d6 Oct 30 21:54:20.775022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.774993 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996"] Oct 30 21:54:20.963974 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.963888 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm"] Oct 30 21:54:20.974018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:20.973991 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kk6pm"] Oct 30 21:54:21.088301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:21.088268 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27"] Oct 30 21:54:21.096198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:21.096172 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-wqg27"] Oct 30 21:54:21.126763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:21.126729 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" podUID="9611497d-715d-438f-98c1-d6b4332ef005" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:54:21.742762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:21.742729 4526 generic.go:356] "Generic (PLEG): container finished" podID="19533110-eb72-49f4-9dc6-7d65d282151a" containerID="de7aead38ec1292cfcee807881b5313c79055d10632846d47292bf104a66532c" exitCode=0 Oct 30 21:54:21.742942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:21.742813 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" event={"ID":"19533110-eb72-49f4-9dc6-7d65d282151a","Type":"ContainerDied","Data":"de7aead38ec1292cfcee807881b5313c79055d10632846d47292bf104a66532c"} Oct 30 21:54:21.742942 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:21.742871 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" event={"ID":"19533110-eb72-49f4-9dc6-7d65d282151a","Type":"ContainerStarted","Data":"20d6c1d82158daa081434a3f5adc87a7671241ff56cd810e3733f5471f6ea5d6"} Oct 30 21:54:21.990941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:21.990861 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52eda2ad-e32a-46e1-ae61-85d9a810f17c" path="/var/lib/kubelet/pods/52eda2ad-e32a-46e1-ae61-85d9a810f17c/volumes" Oct 30 21:54:21.991392 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:21.991378 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" path="/var/lib/kubelet/pods/e8f64983-5fc2-4bcf-a97c-71de7483afa5/volumes" Oct 30 21:54:22.058857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.058819 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" containerID="cri-o://ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f" gracePeriod=45 Oct 30 21:54:22.072225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.072159 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" containerID="cri-o://5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8" gracePeriod=45 Oct 30 21:54:22.132165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.132134 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" podUID="eb8f479d-3076-4853-b172-8a11f4721edd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:54:22.386387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.386362 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:54:22.389634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.389611 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:54:22.474847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.474811 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-home\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-home\") pod \"12bbf005-fe98-432f-a6fc-13e370af59df\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " Oct 30 21:54:22.474847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.474849 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-dshm\") pod \"12bbf005-fe98-432f-a6fc-13e370af59df\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " Oct 30 21:54:22.475117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.474883 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-home\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-home\") pod \"700e8b87-dae4-4881-a139-a090e954220c\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " Oct 30 21:54:22.475117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.474939 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bbf005-fe98-432f-a6fc-13e370af59df-kube-api-access-qklxn\" (UniqueName: \"kubernetes.io/projected/12bbf005-fe98-432f-a6fc-13e370af59df-kube-api-access-qklxn\") pod \"12bbf005-fe98-432f-a6fc-13e370af59df\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " Oct 30 21:54:22.475117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.474966 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700e8b87-dae4-4881-a139-a090e954220c-kube-api-access-b8xjx\" (UniqueName: \"kubernetes.io/projected/700e8b87-dae4-4881-a139-a090e954220c-kube-api-access-b8xjx\") pod \"700e8b87-dae4-4881-a139-a090e954220c\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " Oct 30 21:54:22.475117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475046 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-dshm\") pod \"700e8b87-dae4-4881-a139-a090e954220c\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " Oct 30 21:54:22.475117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475086 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-model-cache\") pod \"12bbf005-fe98-432f-a6fc-13e370af59df\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " Oct 30 21:54:22.475390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475123 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-home" pod "12bbf005-fe98-432f-a6fc-13e370af59df" (UID: "12bbf005-fe98-432f-a6fc-13e370af59df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.475390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475151 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700e8b87-dae4-4881-a139-a090e954220c-tls-certs\" (UniqueName: \"kubernetes.io/secret/700e8b87-dae4-4881-a139-a090e954220c-tls-certs\") pod \"700e8b87-dae4-4881-a139-a090e954220c\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " Oct 30 21:54:22.475390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475183 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-home" pod "700e8b87-dae4-4881-a139-a090e954220c" (UID: "700e8b87-dae4-4881-a139-a090e954220c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.475390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475187 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-model-cache\") pod \"700e8b87-dae4-4881-a139-a090e954220c\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " Oct 30 21:54:22.475390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475319 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-tmp-dir\") pod \"12bbf005-fe98-432f-a6fc-13e370af59df\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " Oct 30 21:54:22.475699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475396 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-model-cache" pod "12bbf005-fe98-432f-a6fc-13e370af59df" (UID: "12bbf005-fe98-432f-a6fc-13e370af59df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.475699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475411 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-tmp-dir\") pod \"700e8b87-dae4-4881-a139-a090e954220c\" (UID: \"700e8b87-dae4-4881-a139-a090e954220c\") " Oct 30 21:54:22.475699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475592 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12bbf005-fe98-432f-a6fc-13e370af59df-tls-certs\" (UniqueName: \"kubernetes.io/secret/12bbf005-fe98-432f-a6fc-13e370af59df-tls-certs\") pod \"12bbf005-fe98-432f-a6fc-13e370af59df\" (UID: \"12bbf005-fe98-432f-a6fc-13e370af59df\") " Oct 30 21:54:22.475699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475609 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-tmp-dir" pod "12bbf005-fe98-432f-a6fc-13e370af59df" (UID: "12bbf005-fe98-432f-a6fc-13e370af59df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.475699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475683 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-tmp-dir" pod "700e8b87-dae4-4881-a139-a090e954220c" (UID: "700e8b87-dae4-4881-a139-a090e954220c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.476015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.475819 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-model-cache" pod "700e8b87-dae4-4881-a139-a090e954220c" (UID: "700e8b87-dae4-4881-a139-a090e954220c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.476581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.476547 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.476581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.476575 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.476762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.476592 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.476762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.476605 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.476762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.476617 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.476762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.476637 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.477972 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.477952 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-dshm" pod "12bbf005-fe98-432f-a6fc-13e370af59df" (UID: "12bbf005-fe98-432f-a6fc-13e370af59df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.478114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.478006 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700e8b87-dae4-4881-a139-a090e954220c-tls-certs" pod "700e8b87-dae4-4881-a139-a090e954220c" (UID: "700e8b87-dae4-4881-a139-a090e954220c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:22.478114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.478096 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700e8b87-dae4-4881-a139-a090e954220c-kube-api-access-b8xjx" pod "700e8b87-dae4-4881-a139-a090e954220c" (UID: "700e8b87-dae4-4881-a139-a090e954220c"). InnerVolumeSpecName "kube-api-access-b8xjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:22.478207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.478125 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bbf005-fe98-432f-a6fc-13e370af59df-kube-api-access-qklxn" pod "12bbf005-fe98-432f-a6fc-13e370af59df" (UID: "12bbf005-fe98-432f-a6fc-13e370af59df"). InnerVolumeSpecName "kube-api-access-qklxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:22.478207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.478132 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-dshm" pod "700e8b87-dae4-4881-a139-a090e954220c" (UID: "700e8b87-dae4-4881-a139-a090e954220c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.478207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.478195 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12bbf005-fe98-432f-a6fc-13e370af59df-tls-certs" pod "12bbf005-fe98-432f-a6fc-13e370af59df" (UID: "12bbf005-fe98-432f-a6fc-13e370af59df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:22.578153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.578107 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12bbf005-fe98-432f-a6fc-13e370af59df-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.578153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.578143 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bbf005-fe98-432f-a6fc-13e370af59df-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.578153 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.578159 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklxn\" (UniqueName: \"kubernetes.io/projected/12bbf005-fe98-432f-a6fc-13e370af59df-kube-api-access-qklxn\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.578506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.578187 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8xjx\" (UniqueName: \"kubernetes.io/projected/700e8b87-dae4-4881-a139-a090e954220c-kube-api-access-b8xjx\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.578506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.578201 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/700e8b87-dae4-4881-a139-a090e954220c-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.578506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.578215 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/700e8b87-dae4-4881-a139-a090e954220c-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.757426 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.757394 4526 generic.go:356] "Generic (PLEG): container finished" podID="12bbf005-fe98-432f-a6fc-13e370af59df" containerID="5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8" exitCode=0 Oct 30 21:54:22.757697 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.757452 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" Oct 30 21:54:22.757697 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.757478 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" event={"ID":"12bbf005-fe98-432f-a6fc-13e370af59df","Type":"ContainerDied","Data":"5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8"} Oct 30 21:54:22.757697 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.757565 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw" event={"ID":"12bbf005-fe98-432f-a6fc-13e370af59df","Type":"ContainerDied","Data":"b407d39ccdf6061eb86d052d74242f6433c65a14044c594c87a98afd3d26f5a0"} Oct 30 21:54:22.757868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.757706 4526 scope.go:122] "RemoveContainer" containerID="5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8" Oct 30 21:54:22.759464 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.759445 4526 generic.go:356] "Generic (PLEG): container finished" podID="700e8b87-dae4-4881-a139-a090e954220c" containerID="ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f" exitCode=0 Oct 30 21:54:22.759656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.759467 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" event={"ID":"700e8b87-dae4-4881-a139-a090e954220c","Type":"ContainerDied","Data":"ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f"} Oct 30 21:54:22.759656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.759516 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" Oct 30 21:54:22.759656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.759540 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" event={"ID":"700e8b87-dae4-4881-a139-a090e954220c","Type":"ContainerDied","Data":"0868b2409c2cc8013d4603811d81b40b34f8740c34d26069ccb8856dcd185a1a"} Oct 30 21:54:22.771684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.771665 4526 scope.go:122] "RemoveContainer" containerID="5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8" Oct 30 21:54:22.772268 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:22.772246 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8\": container with ID starting with 5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8 not found: ID does not exist" containerID="5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8" Oct 30 21:54:22.772316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.772279 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8"} err="failed to get container status \"5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8\": rpc error: code = NotFound desc = could not find container \"5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8\": container with ID starting with 5bdb16311e5a78b6f0caaaecb8762315858f2b1bfcf7aa318480d49b39ac51c8 not found: ID does not exist" Oct 30 21:54:22.772316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.772301 4526 scope.go:122] "RemoveContainer" containerID="ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f" Oct 30 21:54:22.782960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.782935 4526 scope.go:122] "RemoveContainer" containerID="ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f" Oct 30 21:54:22.783319 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:22.783293 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f\": container with ID starting with ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f not found: ID does not exist" containerID="ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f" Oct 30 21:54:22.783388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.783334 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f"} err="failed to get container status \"ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f\": rpc error: code = NotFound desc = could not find container \"ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f\": container with ID starting with ddc47a95c918c563a5e8d039af93c6a5161a7f07aeecec1ad08334ef7702c75f not found: ID does not exist" Oct 30 21:54:22.908859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.908832 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:22.926456 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.926428 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj"] Oct 30 21:54:22.929820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.929799 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj"] Oct 30 21:54:22.982984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.982948 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-openshift-ca\") pod \"19533110-eb72-49f4-9dc6-7d65d282151a\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " Oct 30 21:54:22.983200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.983002 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19533110-eb72-49f4-9dc6-7d65d282151a-kube-api-access-z8b44\" (UniqueName: \"kubernetes.io/projected/19533110-eb72-49f4-9dc6-7d65d282151a-kube-api-access-z8b44\") pod \"19533110-eb72-49f4-9dc6-7d65d282151a\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " Oct 30 21:54:22.983200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.983140 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-xks-ca\") pod \"19533110-eb72-49f4-9dc6-7d65d282151a\" (UID: \"19533110-eb72-49f4-9dc6-7d65d282151a\") " Oct 30 21:54:22.983370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.983349 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-openshift-ca" pod "19533110-eb72-49f4-9dc6-7d65d282151a" (UID: "19533110-eb72-49f4-9dc6-7d65d282151a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:22.983744 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.983672 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.985304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.985278 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-xks-ca" pod "19533110-eb72-49f4-9dc6-7d65d282151a" (UID: "19533110-eb72-49f4-9dc6-7d65d282151a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:22.985407 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:22.985340 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19533110-eb72-49f4-9dc6-7d65d282151a-kube-api-access-z8b44" pod "19533110-eb72-49f4-9dc6-7d65d282151a" (UID: "19533110-eb72-49f4-9dc6-7d65d282151a"). InnerVolumeSpecName "kube-api-access-z8b44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:23.042415 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.042376 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw"] Oct 30 21:54:23.048216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.048189 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/simulator-w6-89c0fd-kserve-58546d97ff-w4kjw"] Oct 30 21:54:23.085319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.085286 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8b44\" (UniqueName: \"kubernetes.io/projected/19533110-eb72-49f4-9dc6-7d65d282151a-kube-api-access-z8b44\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:23.085319 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.085317 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19533110-eb72-49f4-9dc6-7d65d282151a-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:23.320965 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.320916 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/premium-w6-89c0fd-kserve-b8c6d84bb-lkzqj" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 21:54:23.487512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.487466 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz"] Oct 30 21:54:23.487766 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.487736 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" containerID="cri-o://69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa" gracePeriod=30 Oct 30 21:54:23.769694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.769606 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" Oct 30 21:54:23.769694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.769617 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996" event={"ID":"19533110-eb72-49f4-9dc6-7d65d282151a","Type":"ContainerDied","Data":"20d6c1d82158daa081434a3f5adc87a7671241ff56cd810e3733f5471f6ea5d6"} Oct 30 21:54:23.769694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.769674 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d6c1d82158daa081434a3f5adc87a7671241ff56cd810e3733f5471f6ea5d6" Oct 30 21:54:23.985878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.985845 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" path="/var/lib/kubelet/pods/12bbf005-fe98-432f-a6fc-13e370af59df/volumes" Oct 30 21:54:23.986358 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:23.986342 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700e8b87-dae4-4881-a139-a090e954220c" path="/var/lib/kubelet/pods/700e8b87-dae4-4881-a139-a090e954220c/volumes" Oct 30 21:54:26.075156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:26.075119 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68"] Oct 30 21:54:26.124169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:26.124136 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fd552c-mjt68"] Oct 30 21:54:27.986524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:27.986452 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b3346b-3d2b-42b2-9809-0e45272beb36" path="/var/lib/kubelet/pods/f8b3346b-3d2b-42b2-9809-0e45272beb36/volumes" Oct 30 21:54:28.787213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.787188 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:54:28.813034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.812997 4526 generic.go:356] "Generic (PLEG): container finished" podID="8e5c9219-9447-4846-a80c-5adb4466afde" containerID="69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa" exitCode=0 Oct 30 21:54:28.813212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.813044 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" Oct 30 21:54:28.813212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.813043 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" event={"ID":"8e5c9219-9447-4846-a80c-5adb4466afde","Type":"ContainerDied","Data":"69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa"} Oct 30 21:54:28.813212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.813170 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz" event={"ID":"8e5c9219-9447-4846-a80c-5adb4466afde","Type":"ContainerDied","Data":"e13f1c379ce52487ec829237e889ada3fbc4653f80df582e069ec86aa7ba9a35"} Oct 30 21:54:28.813212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.813202 4526 scope.go:122] "RemoveContainer" containerID="69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa" Oct 30 21:54:28.823927 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.823905 4526 scope.go:122] "RemoveContainer" containerID="69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa" Oct 30 21:54:28.824237 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:28.824216 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa\": container with ID starting with 69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa not found: ID does not exist" containerID="69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa" Oct 30 21:54:28.824320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.824248 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa"} err="failed to get container status \"69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa\": rpc error: code = NotFound desc = could not find container \"69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa\": container with ID starting with 69a685989ac223888bbc36f727af08342001296bdd44b1dedf35739496dd96fa not found: ID does not exist" Oct 30 21:54:28.862059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862018 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5c9219-9447-4846-a80c-5adb4466afde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c9219-9447-4846-a80c-5adb4466afde-istiod-ca-cert\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862068 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-kube-api-access-5cxq4\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-kube-api-access-5cxq4\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862111 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e5c9219-9447-4846-a80c-5adb4466afde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c9219-9447-4846-a80c-5adb4466afde-istio-podinfo\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862158 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-data\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862198 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-istio-token\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862261 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-certs\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862410 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-envoy\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862464 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-socket\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862516 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-data" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:28.862672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862408 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5c9219-9447-4846-a80c-5adb4466afde-istiod-ca-cert" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:28.862672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862561 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-credential-socket\") pod \"8e5c9219-9447-4846-a80c-5adb4466afde\" (UID: \"8e5c9219-9447-4846-a80c-5adb4466afde\") " Oct 30 21:54:28.862828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862744 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-socket" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:28.862828 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862768 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-certs" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:28.862930 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.862885 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-credential-socket" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:28.863453 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.863429 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5c9219-9447-4846-a80c-5adb4466afde-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.863622 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.863458 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.863622 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.863474 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.863622 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.863506 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.863622 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.863522 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.864793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.864768 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-kube-api-access-5cxq4" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "kube-api-access-5cxq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:28.864976 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.864955 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-envoy" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:28.865067 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.864974 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-istio-token" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:28.865838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.865817 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e5c9219-9447-4846-a80c-5adb4466afde-istio-podinfo" pod "8e5c9219-9447-4846-a80c-5adb4466afde" (UID: "8e5c9219-9447-4846-a80c-5adb4466afde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:28.965451 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.965420 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5c9219-9447-4846-a80c-5adb4466afde-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.965451 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.965448 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cxq4\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-kube-api-access-5cxq4\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.965451 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.965458 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5c9219-9447-4846-a80c-5adb4466afde-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.965728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:28.965467 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5c9219-9447-4846-a80c-5adb4466afde-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:29.247684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:29.247650 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz"] Oct 30 21:54:29.261044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:29.261015 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-nrspz"] Oct 30 21:54:29.986123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:29.986089 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" path="/var/lib/kubelet/pods/8e5c9219-9447-4846-a80c-5adb4466afde/volumes" Oct 30 21:54:31.126568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:31.126530 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" podUID="9611497d-715d-438f-98c1-d6b4332ef005" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:54:31.652479 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:31.652446 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns"] Oct 30 21:54:31.760982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:31.760944 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8"] Oct 30 21:54:32.132581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:32.132534 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" podUID="eb8f479d-3076-4853-b172-8a11f4721edd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:54:33.487060 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:33.487028 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc"] Oct 30 21:54:33.536280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:33.536246 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9c76ae-h8zcc"] Oct 30 21:54:33.993791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:33.993757 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe093fef-3a8f-4d8e-ab39-9eee95030bed" path="/var/lib/kubelet/pods/fe093fef-3a8f-4d8e-ab39-9eee95030bed/volumes" Oct 30 21:54:35.232017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:35.231981 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l"] Oct 30 21:54:35.232562 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:35.232238 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" containerID="cri-o://dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2" gracePeriod=30 Oct 30 21:54:36.653303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:36.653261 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" podUID="078760c3-3213-4e7d-9a60-935413e9377a" containerName="payload-processing" containerID="cri-o://3d4995ba75df05341eb1f9033fb4ca6b5045239c28517bb05fcf802a58204ca6" gracePeriod=25 Oct 30 21:54:36.761941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:36.761901 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" podUID="d05e1f66-6c14-4e01-b5f0-8e449dbf718a" containerName="payload-pre-processing" containerID="cri-o://47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f" gracePeriod=25 Oct 30 21:54:36.886389 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:36.886341 4526 generic.go:356] "Generic (PLEG): container finished" podID="078760c3-3213-4e7d-9a60-935413e9377a" containerID="3d4995ba75df05341eb1f9033fb4ca6b5045239c28517bb05fcf802a58204ca6" exitCode=0 Oct 30 21:54:36.886570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:36.886401 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" event={"ID":"078760c3-3213-4e7d-9a60-935413e9377a","Type":"ContainerDied","Data":"3d4995ba75df05341eb1f9033fb4ca6b5045239c28517bb05fcf802a58204ca6"} Oct 30 21:54:36.936081 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:36.936056 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:54:37.054747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.054713 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:54:37.085380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.085340 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078760c3-3213-4e7d-9a60-935413e9377a-kube-api-access-w4vxh\" (UniqueName: \"kubernetes.io/projected/078760c3-3213-4e7d-9a60-935413e9377a-kube-api-access-w4vxh\") pod \"078760c3-3213-4e7d-9a60-935413e9377a\" (UID: \"078760c3-3213-4e7d-9a60-935413e9377a\") " Oct 30 21:54:37.085590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.085553 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078760c3-3213-4e7d-9a60-935413e9377a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078760c3-3213-4e7d-9a60-935413e9377a-plugins-config-volume\") pod \"078760c3-3213-4e7d-9a60-935413e9377a\" (UID: \"078760c3-3213-4e7d-9a60-935413e9377a\") " Oct 30 21:54:37.086050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.085986 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078760c3-3213-4e7d-9a60-935413e9377a-plugins-config-volume" pod "078760c3-3213-4e7d-9a60-935413e9377a" (UID: "078760c3-3213-4e7d-9a60-935413e9377a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:37.086613 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.086564 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078760c3-3213-4e7d-9a60-935413e9377a-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:37.088342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.088317 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078760c3-3213-4e7d-9a60-935413e9377a-kube-api-access-w4vxh" pod "078760c3-3213-4e7d-9a60-935413e9377a" (UID: "078760c3-3213-4e7d-9a60-935413e9377a"). InnerVolumeSpecName "kube-api-access-w4vxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:37.187771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.187672 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-kube-api-access-nfxfw\" (UniqueName: \"kubernetes.io/projected/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-kube-api-access-nfxfw\") pod \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\" (UID: \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\") " Oct 30 21:54:37.187771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.187755 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-plugins-config-volume\") pod \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\" (UID: \"d05e1f66-6c14-4e01-b5f0-8e449dbf718a\") " Oct 30 21:54:37.188120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.188098 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-plugins-config-volume" pod "d05e1f66-6c14-4e01-b5f0-8e449dbf718a" (UID: "d05e1f66-6c14-4e01-b5f0-8e449dbf718a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:37.188367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.188351 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:37.188441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.188368 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4vxh\" (UniqueName: \"kubernetes.io/projected/078760c3-3213-4e7d-9a60-935413e9377a-kube-api-access-w4vxh\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:37.190117 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.190089 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-kube-api-access-nfxfw" pod "d05e1f66-6c14-4e01-b5f0-8e449dbf718a" (UID: "d05e1f66-6c14-4e01-b5f0-8e449dbf718a"). InnerVolumeSpecName "kube-api-access-nfxfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:37.290083 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.290047 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxfw\" (UniqueName: \"kubernetes.io/projected/d05e1f66-6c14-4e01-b5f0-8e449dbf718a-kube-api-access-nfxfw\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:37.895507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.895453 4526 generic.go:356] "Generic (PLEG): container finished" podID="d05e1f66-6c14-4e01-b5f0-8e449dbf718a" containerID="47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f" exitCode=0 Oct 30 21:54:37.895964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.895640 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" event={"ID":"d05e1f66-6c14-4e01-b5f0-8e449dbf718a","Type":"ContainerDied","Data":"47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f"} Oct 30 21:54:37.895964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.895696 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" event={"ID":"d05e1f66-6c14-4e01-b5f0-8e449dbf718a","Type":"ContainerDied","Data":"024a64b5fcc76de2e16beb5b52056b333d3e8a26de07e12646d695e9bc37fa95"} Oct 30 21:54:37.895964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.895726 4526 scope.go:122] "RemoveContainer" containerID="47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f" Oct 30 21:54:37.895964 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.895882 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8" Oct 30 21:54:37.901606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.901578 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" event={"ID":"078760c3-3213-4e7d-9a60-935413e9377a","Type":"ContainerDied","Data":"79e316e3e197fd6884a4cf31813f2dedff89227dc2cf3d422379abac59c59487"} Oct 30 21:54:37.901808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.901478 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns" Oct 30 21:54:37.914162 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.914143 4526 scope.go:122] "RemoveContainer" containerID="47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f" Oct 30 21:54:37.914724 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:37.914700 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f\": container with ID starting with 47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f not found: ID does not exist" containerID="47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f" Oct 30 21:54:37.914833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.914729 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f"} err="failed to get container status \"47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f\": rpc error: code = NotFound desc = could not find container \"47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f\": container with ID starting with 47fb6749bda43767f909c46317ba5db27a7d8d4c4d36dd8c30304767ea2cb31f not found: ID does not exist" Oct 30 21:54:37.914833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:37.914747 4526 scope.go:122] "RemoveContainer" containerID="3d4995ba75df05341eb1f9033fb4ca6b5045239c28517bb05fcf802a58204ca6" Oct 30 21:54:38.079549 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:38.079507 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8"] Oct 30 21:54:38.119736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:38.119709 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-2ace1d-5456899699-jbtk8"] Oct 30 21:54:38.245331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:38.245221 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns"] Oct 30 21:54:38.249011 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:38.248969 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-2ace1d-77f8c57d55-zd8ns"] Oct 30 21:54:39.813507 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:39.813459 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:39.813940 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:39.813555 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:39.989059 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:39.988929 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078760c3-3213-4e7d-9a60-935413e9377a" path="/var/lib/kubelet/pods/078760c3-3213-4e7d-9a60-935413e9377a/volumes" Oct 30 21:54:39.989551 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:39.989528 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05e1f66-6c14-4e01-b5f0-8e449dbf718a" path="/var/lib/kubelet/pods/d05e1f66-6c14-4e01-b5f0-8e449dbf718a/volumes" Oct 30 21:54:40.204554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.204522 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996"] Oct 30 21:54:40.254640 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.254607 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-2ace1d-zh996"] Oct 30 21:54:40.569960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.569933 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:54:40.593574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.593549 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:54:40.597725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.597704 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:54:40.736856 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.736763 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8f479d-3076-4853-b172-8a11f4721edd-kube-api-access-n5rn5\" (UniqueName: \"kubernetes.io/projected/eb8f479d-3076-4853-b172-8a11f4721edd-kube-api-access-n5rn5\") pod \"eb8f479d-3076-4853-b172-8a11f4721edd\" (UID: \"eb8f479d-3076-4853-b172-8a11f4721edd\") " Oct 30 21:54:40.736856 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.736852 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-token\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-token\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.736892 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-podinfo\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.736923 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8f479d-3076-4853-b172-8a11f4721edd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f479d-3076-4853-b172-8a11f4721edd-plugins-config-volume\") pod \"eb8f479d-3076-4853-b172-8a11f4721edd\" (UID: \"eb8f479d-3076-4853-b172-8a11f4721edd\") " Oct 30 21:54:40.737115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.736974 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9611497d-715d-438f-98c1-d6b4332ef005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9611497d-715d-438f-98c1-d6b4332ef005-plugins-config-volume\") pod \"9611497d-715d-438f-98c1-d6b4332ef005\" (UID: \"9611497d-715d-438f-98c1-d6b4332ef005\") " Oct 30 21:54:40.737115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737014 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-envoy\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737062 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-data\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737127 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-socket\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737179 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istiod-ca-cert\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737238 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-certs\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737270 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9611497d-715d-438f-98c1-d6b4332ef005-kube-api-access-vs9nm\" (UniqueName: \"kubernetes.io/projected/9611497d-715d-438f-98c1-d6b4332ef005-kube-api-access-vs9nm\") pod \"9611497d-715d-438f-98c1-d6b4332ef005\" (UID: \"9611497d-715d-438f-98c1-d6b4332ef005\") " Oct 30 21:54:40.737388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737332 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-credential-socket\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737428 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-data" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:40.737891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737523 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-kube-api-access-49jtv\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-kube-api-access-49jtv\") pod \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\" (UID: \"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf\") " Oct 30 21:54:40.737891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737773 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istiod-ca-cert" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:40.737891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.737860 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9611497d-715d-438f-98c1-d6b4332ef005-plugins-config-volume" pod "9611497d-715d-438f-98c1-d6b4332ef005" (UID: "9611497d-715d-438f-98c1-d6b4332ef005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:40.738170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.738146 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-socket" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:40.739266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.738282 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8f479d-3076-4853-b172-8a11f4721edd-plugins-config-volume" pod "eb8f479d-3076-4853-b172-8a11f4721edd" (UID: "eb8f479d-3076-4853-b172-8a11f4721edd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:40.739266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.738878 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb8f479d-3076-4853-b172-8a11f4721edd-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.739266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.738902 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9611497d-715d-438f-98c1-d6b4332ef005-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.739266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.738918 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.739266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.738933 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.739266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.738947 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.739266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.739146 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-credential-socket" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:40.739266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.739239 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-certs" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:40.740783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.740755 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-podinfo" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:40.740920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.740894 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-envoy" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:40.742222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.742060 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-token" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:40.742845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.742808 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-kube-api-access-49jtv" pod "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" (UID: "a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf"). InnerVolumeSpecName "kube-api-access-49jtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:40.743239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.743203 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9611497d-715d-438f-98c1-d6b4332ef005-kube-api-access-vs9nm" pod "9611497d-715d-438f-98c1-d6b4332ef005" (UID: "9611497d-715d-438f-98c1-d6b4332ef005"). InnerVolumeSpecName "kube-api-access-vs9nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:40.743351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.743263 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8f479d-3076-4853-b172-8a11f4721edd-kube-api-access-n5rn5" pod "eb8f479d-3076-4853-b172-8a11f4721edd" (UID: "eb8f479d-3076-4853-b172-8a11f4721edd"). InnerVolumeSpecName "kube-api-access-n5rn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:40.841005 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.840969 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.841005 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.841008 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs9nm\" (UniqueName: \"kubernetes.io/projected/9611497d-715d-438f-98c1-d6b4332ef005-kube-api-access-vs9nm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.841425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.841024 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.841425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.841039 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jtv\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-kube-api-access-49jtv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.841425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.841056 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rn5\" (UniqueName: \"kubernetes.io/projected/eb8f479d-3076-4853-b172-8a11f4721edd-kube-api-access-n5rn5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.841425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.841072 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.841425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.841086 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:40.841425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:40.841099 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:41.026211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.026127 4526 generic.go:356] "Generic (PLEG): container finished" podID="9611497d-715d-438f-98c1-d6b4332ef005" containerID="26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d" exitCode=0 Oct 30 21:54:41.026751 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.026722 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" event={"ID":"9611497d-715d-438f-98c1-d6b4332ef005","Type":"ContainerDied","Data":"26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d"} Oct 30 21:54:41.026960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.026945 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" event={"ID":"9611497d-715d-438f-98c1-d6b4332ef005","Type":"ContainerDied","Data":"0b5d0957f3e3ca09926a45af224db86772554a7284f9f13ea91e542151aa45fa"} Oct 30 21:54:41.027072 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.027059 4526 scope.go:122] "RemoveContainer" containerID="26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d" Oct 30 21:54:41.027317 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.027303 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd" Oct 30 21:54:41.033715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.033668 4526 generic.go:356] "Generic (PLEG): container finished" podID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerID="dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2" exitCode=0 Oct 30 21:54:41.033834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.033772 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" Oct 30 21:54:41.033834 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.033786 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" event={"ID":"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf","Type":"ContainerDied","Data":"dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2"} Oct 30 21:54:41.033952 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.033868 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l" event={"ID":"a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf","Type":"ContainerDied","Data":"b1bafdbf12a9f94d0ac476303efe07736f150eaad1c3c9aaa88a7e7f5a0de62a"} Oct 30 21:54:41.035973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.035932 4526 generic.go:356] "Generic (PLEG): container finished" podID="eb8f479d-3076-4853-b172-8a11f4721edd" containerID="da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846" exitCode=0 Oct 30 21:54:41.036110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.036033 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" event={"ID":"eb8f479d-3076-4853-b172-8a11f4721edd","Type":"ContainerDied","Data":"da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846"} Oct 30 21:54:41.036110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.036093 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" event={"ID":"eb8f479d-3076-4853-b172-8a11f4721edd","Type":"ContainerDied","Data":"4838062a167c09fb9ec674b702ef47e3663f3f7447c15b4a441d7d807dc6f30b"} Oct 30 21:54:41.036295 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.036189 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt" Oct 30 21:54:41.049654 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.049553 4526 scope.go:122] "RemoveContainer" containerID="26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d" Oct 30 21:54:41.049984 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:41.049951 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d\": container with ID starting with 26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d not found: ID does not exist" containerID="26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d" Oct 30 21:54:41.050108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.049994 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d"} err="failed to get container status \"26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d\": rpc error: code = NotFound desc = could not find container \"26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d\": container with ID starting with 26ecef3350f634621d7d845f1cb31aaec596f98af93c8234fb6c583571e81e1d not found: ID does not exist" Oct 30 21:54:41.050108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.050021 4526 scope.go:122] "RemoveContainer" containerID="dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2" Oct 30 21:54:41.065803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.065701 4526 scope.go:122] "RemoveContainer" containerID="dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2" Oct 30 21:54:41.066215 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:41.066192 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2\": container with ID starting with dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2 not found: ID does not exist" containerID="dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2" Oct 30 21:54:41.066326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.066227 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2"} err="failed to get container status \"dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2\": rpc error: code = NotFound desc = could not find container \"dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2\": container with ID starting with dc4b48ea0b33721caf989c9e659ae7f93b23246a066698e448a011e6376ff1f2 not found: ID does not exist" Oct 30 21:54:41.066326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.066250 4526 scope.go:122] "RemoveContainer" containerID="da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846" Oct 30 21:54:41.078839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.078813 4526 scope.go:122] "RemoveContainer" containerID="da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846" Oct 30 21:54:41.079189 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:41.079167 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846\": container with ID starting with da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846 not found: ID does not exist" containerID="da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846" Oct 30 21:54:41.079274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.079196 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846"} err="failed to get container status \"da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846\": rpc error: code = NotFound desc = could not find container \"da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846\": container with ID starting with da518f8bfd97cc6327b734962bca6822f610767d5b2c64f9bbbb8ebd152cd846 not found: ID does not exist" Oct 30 21:54:41.195705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.195671 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt"] Oct 30 21:54:41.200135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.200105 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-89c0fd-78cdfddb47-f58wt"] Oct 30 21:54:41.312112 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.312083 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd"] Oct 30 21:54:41.317202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.317176 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-89c0fd-655b9c447f-zvkkd"] Oct 30 21:54:41.428251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.428219 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l"] Oct 30 21:54:41.433293 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.433270 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-hrq9l"] Oct 30 21:54:41.985632 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.985600 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19533110-eb72-49f4-9dc6-7d65d282151a" path="/var/lib/kubelet/pods/19533110-eb72-49f4-9dc6-7d65d282151a/volumes" Oct 30 21:54:41.986008 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.985994 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9611497d-715d-438f-98c1-d6b4332ef005" path="/var/lib/kubelet/pods/9611497d-715d-438f-98c1-d6b4332ef005/volumes" Oct 30 21:54:41.986409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.986397 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" path="/var/lib/kubelet/pods/a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf/volumes" Oct 30 21:54:41.987063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:41.987048 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8f479d-3076-4853-b172-8a11f4721edd" path="/var/lib/kubelet/pods/eb8f479d-3076-4853-b172-8a11f4721edd/volumes" Oct 30 21:54:42.501545 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:54:42.501477 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:54:43.065223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065194 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr"] Oct 30 21:54:43.065770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065755 4526 cpu_manager.go:414] "containerMap: removing container" podUID="d05e1f66-6c14-4e01-b5f0-8e449dbf718a" containerName="payload-pre-processing" Oct 30 21:54:43.065816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065771 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05e1f66-6c14-4e01-b5f0-8e449dbf718a" containerName="payload-pre-processing" Oct 30 21:54:43.065816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065788 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" Oct 30 21:54:43.065816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065793 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" Oct 30 21:54:43.065816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065803 4526 cpu_manager.go:414] "containerMap: removing container" podUID="9611497d-715d-438f-98c1-d6b4332ef005" containerName="payload-pre-processing" Oct 30 21:54:43.065816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065809 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9611497d-715d-438f-98c1-d6b4332ef005" containerName="payload-pre-processing" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065821 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065826 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065836 4526 cpu_manager.go:414] "containerMap: removing container" podUID="19533110-eb72-49f4-9dc6-7d65d282151a" containerName="revoke-keys" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065841 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19533110-eb72-49f4-9dc6-7d65d282151a" containerName="revoke-keys" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065852 4526 cpu_manager.go:414] "containerMap: removing container" podUID="eb8f479d-3076-4853-b172-8a11f4721edd" containerName="payload-processing" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065856 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8f479d-3076-4853-b172-8a11f4721edd" containerName="payload-processing" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065863 4526 cpu_manager.go:414] "containerMap: removing container" podUID="52eda2ad-e32a-46e1-ae61-85d9a810f17c" containerName="discovery" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065867 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52eda2ad-e32a-46e1-ae61-85d9a810f17c" containerName="discovery" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065873 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065877 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6093f34-95cd-453d-bc0e-8f7bc5ecfcec" containerName="istio-proxy" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065884 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065888 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065895 4526 cpu_manager.go:414] "containerMap: removing container" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065899 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065906 4526 cpu_manager.go:414] "containerMap: removing container" podUID="078760c3-3213-4e7d-9a60-935413e9377a" containerName="payload-processing" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065911 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078760c3-3213-4e7d-9a60-935413e9377a" containerName="payload-processing" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065923 4526 cpu_manager.go:414] "containerMap: removing container" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" Oct 30 21:54:43.065973 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.065928 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066008 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bbf005-fe98-432f-a6fc-13e370af59df" containerName="main" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066022 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05e1f66-6c14-4e01-b5f0-8e449dbf718a" containerName="payload-pre-processing" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066030 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8644e88-b41d-4e2e-ae13-b8d6f2b8bfcf" containerName="istio-proxy" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066038 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="9611497d-715d-438f-98c1-d6b4332ef005" containerName="payload-pre-processing" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066044 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5c9219-9447-4846-a80c-5adb4466afde" containerName="istio-proxy" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066050 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="19533110-eb72-49f4-9dc6-7d65d282151a" containerName="revoke-keys" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066055 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8f479d-3076-4853-b172-8a11f4721edd" containerName="payload-processing" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066059 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="52eda2ad-e32a-46e1-ae61-85d9a810f17c" containerName="discovery" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066064 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f64983-5fc2-4bcf-a97c-71de7483afa5" containerName="istio-proxy" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066070 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="700e8b87-dae4-4881-a139-a090e954220c" containerName="main" Oct 30 21:54:43.066526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.066075 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="078760c3-3213-4e7d-9a60-935413e9377a" containerName="payload-processing" Oct 30 21:54:43.070941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.070921 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.132137 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.132115 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:43.174975 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.174944 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.175128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.174982 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.175128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.175089 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46dd2\" (UniqueName: \"kubernetes.io/projected/da690afa-ea30-4227-a106-c590b5a626ac-kube-api-access-46dd2\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.176754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.176734 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:43.176812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.176763 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:43.180347 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.180325 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr"] Oct 30 21:54:43.231592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.231567 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:54:43.275815 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.275778 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.275984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.275823 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.275984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.275906 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46dd2\" (UniqueName: \"kubernetes.io/projected/da690afa-ea30-4227-a106-c590b5a626ac-kube-api-access-46dd2\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.276570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.276552 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.278347 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.278330 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.337206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.337137 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46dd2\" (UniqueName: \"kubernetes.io/projected/da690afa-ea30-4227-a106-c590b5a626ac-kube-api-access-46dd2\") pod \"maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.384620 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.384588 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:43.765134 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:43.765104 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr"] Oct 30 21:54:43.765709 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:54:43.765679 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda690afa_ea30_4227_a106_c590b5a626ac.slice/crio-7c253558bbc58adb6370610a1169181c83dd2e0556a2038a3db0307d78d36f41 WatchSource:0}: Error finding container 7c253558bbc58adb6370610a1169181c83dd2e0556a2038a3db0307d78d36f41: Status 404 returned error can't find the container with id 7c253558bbc58adb6370610a1169181c83dd2e0556a2038a3db0307d78d36f41 Oct 30 21:54:44.059820 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:44.059783 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" event={"ID":"da690afa-ea30-4227-a106-c590b5a626ac","Type":"ContainerStarted","Data":"7c253558bbc58adb6370610a1169181c83dd2e0556a2038a3db0307d78d36f41"} Oct 30 21:54:45.075749 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:45.075694 4526 generic.go:356] "Generic (PLEG): container finished" podID="da690afa-ea30-4227-a106-c590b5a626ac" containerID="2494766b655a54889d38313dc1aeb7cc56ff24f62a77a6e255154d47f9068f4d" exitCode=0 Oct 30 21:54:45.076260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:45.075799 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" event={"ID":"da690afa-ea30-4227-a106-c590b5a626ac","Type":"ContainerDied","Data":"2494766b655a54889d38313dc1aeb7cc56ff24f62a77a6e255154d47f9068f4d"} Oct 30 21:54:46.256900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.256876 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:46.411357 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.411277 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-openshift-ca\") pod \"da690afa-ea30-4227-a106-c590b5a626ac\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " Oct 30 21:54:46.411512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.411377 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-xks-ca\") pod \"da690afa-ea30-4227-a106-c590b5a626ac\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " Oct 30 21:54:46.411593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.411575 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da690afa-ea30-4227-a106-c590b5a626ac-kube-api-access-46dd2\" (UniqueName: \"kubernetes.io/projected/da690afa-ea30-4227-a106-c590b5a626ac-kube-api-access-46dd2\") pod \"da690afa-ea30-4227-a106-c590b5a626ac\" (UID: \"da690afa-ea30-4227-a106-c590b5a626ac\") " Oct 30 21:54:46.411660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.411638 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-openshift-ca" pod "da690afa-ea30-4227-a106-c590b5a626ac" (UID: "da690afa-ea30-4227-a106-c590b5a626ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:46.412184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.412147 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.413684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.413661 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-xks-ca" pod "da690afa-ea30-4227-a106-c590b5a626ac" (UID: "da690afa-ea30-4227-a106-c590b5a626ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:46.413853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.413837 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da690afa-ea30-4227-a106-c590b5a626ac-kube-api-access-46dd2" pod "da690afa-ea30-4227-a106-c590b5a626ac" (UID: "da690afa-ea30-4227-a106-c590b5a626ac"). InnerVolumeSpecName "kube-api-access-46dd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:46.513920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.513886 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da690afa-ea30-4227-a106-c590b5a626ac-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.513920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:46.513915 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46dd2\" (UniqueName: \"kubernetes.io/projected/da690afa-ea30-4227-a106-c590b5a626ac-kube-api-access-46dd2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:47.096801 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:47.096776 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" Oct 30 21:54:47.096951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:47.096815 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr" event={"ID":"da690afa-ea30-4227-a106-c590b5a626ac","Type":"ContainerDied","Data":"7c253558bbc58adb6370610a1169181c83dd2e0556a2038a3db0307d78d36f41"} Oct 30 21:54:47.096951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:47.096862 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c253558bbc58adb6370610a1169181c83dd2e0556a2038a3db0307d78d36f41" Oct 30 21:54:47.571791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:47.571756 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4"] Oct 30 21:54:47.602578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:47.602546 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5"] Oct 30 21:54:47.927325 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:47.927244 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt"] Oct 30 21:54:47.927501 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:47.927439 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" containerID="cri-o://86bcd892373262f756df14a998a1bb7d137b616332366ae2158de3d20f60208f" gracePeriod=30 Oct 30 21:54:49.759662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.759631 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg"] Oct 30 21:54:49.760262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.760248 4526 cpu_manager.go:414] "containerMap: removing container" podUID="da690afa-ea30-4227-a106-c590b5a626ac" containerName="revoke-keys" Oct 30 21:54:49.760311 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.760263 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da690afa-ea30-4227-a106-c590b5a626ac" containerName="revoke-keys" Oct 30 21:54:49.760356 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.760347 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="da690afa-ea30-4227-a106-c590b5a626ac" containerName="revoke-keys" Oct 30 21:54:49.763563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.763543 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:49.816683 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.816649 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:54:49.820537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.820510 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn"] Oct 30 21:54:49.826857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.826822 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:49.858803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.858764 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrl7\" (UniqueName: \"kubernetes.io/projected/05f408a1-4cc4-455d-ae9b-07b577205888-kube-api-access-9xrl7\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:49.859031 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.858883 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:49.859031 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.858979 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:49.866446 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.866413 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:49.866691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.866433 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:49.866691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.866439 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:49.872806 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.872770 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg"] Oct 30 21:54:49.892367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.892324 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn"] Oct 30 21:54:49.960248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.960209 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6md\" (UniqueName: \"kubernetes.io/projected/3bf32042-919c-4a6d-b379-676feb9106b7-kube-api-access-lx6md\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:49.960428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.960270 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:49.960428 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.960390 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:49.960603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.960446 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrl7\" (UniqueName: \"kubernetes.io/projected/05f408a1-4cc4-455d-ae9b-07b577205888-kube-api-access-9xrl7\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:49.960603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.960482 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:49.960603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.960590 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:49.961379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.961360 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:49.963173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:49.963143 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:50.061644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.061610 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6md\" (UniqueName: \"kubernetes.io/projected/3bf32042-919c-4a6d-b379-676feb9106b7-kube-api-access-lx6md\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:50.061853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.061653 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:50.061853 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.061679 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:50.062380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.062351 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:50.064410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.064381 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:50.066521 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.066504 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:50.079786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.079759 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrl7\" (UniqueName: \"kubernetes.io/projected/05f408a1-4cc4-455d-ae9b-07b577205888-kube-api-access-9xrl7\") pod \"maas-api-revoke-sub-b96f5c85-5w2jg\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:50.118894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.118864 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6md\" (UniqueName: \"kubernetes.io/projected/3bf32042-919c-4a6d-b379-676feb9106b7-kube-api-access-lx6md\") pod \"maas-api-revoke-sub-8b9f1b94-q72kn\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:50.194406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.194367 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:54:50.204035 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.203995 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:50.378772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.378675 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:50.600662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.600630 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn"] Oct 30 21:54:50.601347 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:54:50.601314 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf32042_919c_4a6d_b379_676feb9106b7.slice/crio-0886b9b69cfdf02a93975a74e39929072ffea106be75a91e562d9e7b39abfcc1 WatchSource:0}: Error finding container 0886b9b69cfdf02a93975a74e39929072ffea106be75a91e562d9e7b39abfcc1: Status 404 returned error can't find the container with id 0886b9b69cfdf02a93975a74e39929072ffea106be75a91e562d9e7b39abfcc1 Oct 30 21:54:50.770256 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:50.770234 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg"] Oct 30 21:54:50.771648 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:54:50.771620 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f408a1_4cc4_455d_ae9b_07b577205888.slice/crio-253c5b49d866eb4f21b96020f48774b908403d2e506c9d74ab75a2f5dfb4e8f2 WatchSource:0}: Error finding container 253c5b49d866eb4f21b96020f48774b908403d2e506c9d74ab75a2f5dfb4e8f2: Status 404 returned error can't find the container with id 253c5b49d866eb4f21b96020f48774b908403d2e506c9d74ab75a2f5dfb4e8f2 Oct 30 21:54:51.132158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:51.132123 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" event={"ID":"05f408a1-4cc4-455d-ae9b-07b577205888","Type":"ContainerStarted","Data":"253c5b49d866eb4f21b96020f48774b908403d2e506c9d74ab75a2f5dfb4e8f2"} Oct 30 21:54:51.133845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:51.133820 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" event={"ID":"3bf32042-919c-4a6d-b379-676feb9106b7","Type":"ContainerStarted","Data":"175e72b3ce3ab77ae8599d900a1f29d764dbab2c47356ea04173ae958d8efb95"} Oct 30 21:54:51.133954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:51.133863 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" event={"ID":"3bf32042-919c-4a6d-b379-676feb9106b7","Type":"ContainerStarted","Data":"0886b9b69cfdf02a93975a74e39929072ffea106be75a91e562d9e7b39abfcc1"} Oct 30 21:54:51.247936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:51.247881 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" podStartSLOduration=2.2478637790000002 podStartE2EDuration="2.247863779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:51.247284463 +0000 UTC m=+2281.380657663" watchObservedRunningTime="2026-10-30 21:54:51.247863779 +0000 UTC m=+2281.381236977" Oct 30 21:54:51.880084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:51.880006 4526 scope.go:122] "RemoveContainer" containerID="9c152563ac8522e5147f39da5e487767f32515e1ec518570e6db2c6163a2923e" Oct 30 21:54:52.148051 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:52.147957 4526 generic.go:356] "Generic (PLEG): container finished" podID="05f408a1-4cc4-455d-ae9b-07b577205888" containerID="b099f4a414908e1ab0589b0a322880914453857e88591857ac09f3b71186bc99" exitCode=0 Oct 30 21:54:52.148051 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:52.148012 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" event={"ID":"05f408a1-4cc4-455d-ae9b-07b577205888","Type":"ContainerDied","Data":"b099f4a414908e1ab0589b0a322880914453857e88591857ac09f3b71186bc99"} Oct 30 21:54:52.149754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:52.149728 4526 generic.go:356] "Generic (PLEG): container finished" podID="3bf32042-919c-4a6d-b379-676feb9106b7" containerID="175e72b3ce3ab77ae8599d900a1f29d764dbab2c47356ea04173ae958d8efb95" exitCode=0 Oct 30 21:54:52.149855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:52.149799 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" event={"ID":"3bf32042-919c-4a6d-b379-676feb9106b7","Type":"ContainerDied","Data":"175e72b3ce3ab77ae8599d900a1f29d764dbab2c47356ea04173ae958d8efb95"} Oct 30 21:54:52.268472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:52.268431 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb"] Oct 30 21:54:52.317965 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:52.317933 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-013fb2-rwgzb"] Oct 30 21:54:53.161686 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.161650 4526 generic.go:356] "Generic (PLEG): container finished" podID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerID="86bcd892373262f756df14a998a1bb7d137b616332366ae2158de3d20f60208f" exitCode=0 Oct 30 21:54:53.162155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.161711 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" event={"ID":"f4dc33b1-e449-4b7f-b1eb-10adff633529","Type":"ContainerDied","Data":"86bcd892373262f756df14a998a1bb7d137b616332366ae2158de3d20f60208f"} Oct 30 21:54:53.162155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.161795 4526 scope.go:122] "RemoveContainer" containerID="fdb610614e5912c4863a59bc3572df620dda014d9aab572e68590ed7bca497c1" Oct 30 21:54:53.221471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.221412 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:54:53.311954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.311925 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dc33b1-e449-4b7f-b1eb-10adff633529-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc33b1-e449-4b7f-b1eb-10adff633529-istiod-ca-cert\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.311974 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-envoy\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312015 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-certs\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312063 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-data\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312124 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312091 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-podinfo\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312259 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-kube-api-access-4x74p\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-kube-api-access-4x74p\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312303 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-credential-socket\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312342 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-token\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-token\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312432 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-socket\") pod \"f4dc33b1-e449-4b7f-b1eb-10adff633529\" (UID: \"f4dc33b1-e449-4b7f-b1eb-10adff633529\") " Oct 30 21:54:53.312931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312437 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-certs" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:53.312931 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.312866 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-data" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:53.313288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.313097 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-socket" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:53.313288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.313195 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.313288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.313219 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.313288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.313233 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.313288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.313241 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-credential-socket" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:53.313769 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.313478 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dc33b1-e449-4b7f-b1eb-10adff633529-istiod-ca-cert" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:53.316186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.316160 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-kube-api-access-4x74p" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "kube-api-access-4x74p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:53.318653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.318625 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-envoy" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:53.318653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.318636 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-podinfo" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:53.318826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.318652 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-token" pod "f4dc33b1-e449-4b7f-b1eb-10adff633529" (UID: "f4dc33b1-e449-4b7f-b1eb-10adff633529"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:53.345253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.345225 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:53.350717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.350691 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:53.413988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.413958 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f408a1-4cc4-455d-ae9b-07b577205888-kube-api-access-9xrl7\" (UniqueName: \"kubernetes.io/projected/05f408a1-4cc4-455d-ae9b-07b577205888-kube-api-access-9xrl7\") pod \"05f408a1-4cc4-455d-ae9b-07b577205888\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " Oct 30 21:54:53.414158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.414082 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-openshift-ca\") pod \"05f408a1-4cc4-455d-ae9b-07b577205888\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " Oct 30 21:54:53.414158 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.414133 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf32042-919c-4a6d-b379-676feb9106b7-kube-api-access-lx6md\" (UniqueName: \"kubernetes.io/projected/3bf32042-919c-4a6d-b379-676feb9106b7-kube-api-access-lx6md\") pod \"3bf32042-919c-4a6d-b379-676feb9106b7\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " Oct 30 21:54:53.414248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.414166 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-xks-ca\") pod \"3bf32042-919c-4a6d-b379-676feb9106b7\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " Oct 30 21:54:53.414305 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.414273 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-xks-ca\") pod \"05f408a1-4cc4-455d-ae9b-07b577205888\" (UID: \"05f408a1-4cc4-455d-ae9b-07b577205888\") " Oct 30 21:54:53.414362 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.414316 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-openshift-ca\") pod \"3bf32042-919c-4a6d-b379-676feb9106b7\" (UID: \"3bf32042-919c-4a6d-b379-676feb9106b7\") " Oct 30 21:54:53.414568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.414534 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-openshift-ca" pod "05f408a1-4cc4-455d-ae9b-07b577205888" (UID: "05f408a1-4cc4-455d-ae9b-07b577205888"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:53.414956 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.414887 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-openshift-ca" pod "3bf32042-919c-4a6d-b379-676feb9106b7" (UID: "3bf32042-919c-4a6d-b379-676feb9106b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:53.415121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.415100 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x74p\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-kube-api-access-4x74p\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.415185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.415129 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.415185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.415144 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.415185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.415157 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dc33b1-e449-4b7f-b1eb-10adff633529-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.415342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.415172 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.415342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.415204 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.415342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.415219 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dc33b1-e449-4b7f-b1eb-10adff633529-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.416677 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.416650 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-xks-ca" pod "3bf32042-919c-4a6d-b379-676feb9106b7" (UID: "3bf32042-919c-4a6d-b379-676feb9106b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:53.417114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.417091 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-xks-ca" pod "05f408a1-4cc4-455d-ae9b-07b577205888" (UID: "05f408a1-4cc4-455d-ae9b-07b577205888"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:53.417233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.417214 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf32042-919c-4a6d-b379-676feb9106b7-kube-api-access-lx6md" pod "3bf32042-919c-4a6d-b379-676feb9106b7" (UID: "3bf32042-919c-4a6d-b379-676feb9106b7"). InnerVolumeSpecName "kube-api-access-lx6md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:53.417453 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.417434 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f408a1-4cc4-455d-ae9b-07b577205888-kube-api-access-9xrl7" pod "05f408a1-4cc4-455d-ae9b-07b577205888" (UID: "05f408a1-4cc4-455d-ae9b-07b577205888"). InnerVolumeSpecName "kube-api-access-9xrl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:53.516438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.516352 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f408a1-4cc4-455d-ae9b-07b577205888-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.516438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.516387 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.516438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.516401 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xrl7\" (UniqueName: \"kubernetes.io/projected/05f408a1-4cc4-455d-ae9b-07b577205888-kube-api-access-9xrl7\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.516438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.516414 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6md\" (UniqueName: \"kubernetes.io/projected/3bf32042-919c-4a6d-b379-676feb9106b7-kube-api-access-lx6md\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.516438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.516428 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf32042-919c-4a6d-b379-676feb9106b7-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:53.985734 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:53.985703 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7d2f5d-e789-46c6-a38a-f4c960126d85" path="/var/lib/kubelet/pods/ff7d2f5d-e789-46c6-a38a-f4c960126d85/volumes" Oct 30 21:54:54.169901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.169873 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" Oct 30 21:54:54.170363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.169902 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg" event={"ID":"05f408a1-4cc4-455d-ae9b-07b577205888","Type":"ContainerDied","Data":"253c5b49d866eb4f21b96020f48774b908403d2e506c9d74ab75a2f5dfb4e8f2"} Oct 30 21:54:54.170363 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.169951 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="253c5b49d866eb4f21b96020f48774b908403d2e506c9d74ab75a2f5dfb4e8f2" Oct 30 21:54:54.171592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.171557 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" event={"ID":"f4dc33b1-e449-4b7f-b1eb-10adff633529","Type":"ContainerDied","Data":"98d5af3b909db08c5bb8b4af06c9123cc62a9129e81feab08eee58fcaa06f9cb"} Oct 30 21:54:54.171740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.171564 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt" Oct 30 21:54:54.171740 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.171615 4526 scope.go:122] "RemoveContainer" containerID="86bcd892373262f756df14a998a1bb7d137b616332366ae2158de3d20f60208f" Oct 30 21:54:54.173894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.173867 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" event={"ID":"3bf32042-919c-4a6d-b379-676feb9106b7","Type":"ContainerDied","Data":"0886b9b69cfdf02a93975a74e39929072ffea106be75a91e562d9e7b39abfcc1"} Oct 30 21:54:54.173894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.173885 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn" Oct 30 21:54:54.174088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.173937 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0886b9b69cfdf02a93975a74e39929072ffea106be75a91e562d9e7b39abfcc1" Oct 30 21:54:54.394532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.394499 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt"] Oct 30 21:54:54.398178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.398152 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-2ace1d-openshift-default-78d5c9bf57-dfwkt"] Oct 30 21:54:54.580592 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.580564 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g"] Oct 30 21:54:54.678772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:54.678694 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv"] Oct 30 21:54:55.985436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:55.985399 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" path="/var/lib/kubelet/pods/f4dc33b1-e449-4b7f-b1eb-10adff633529/volumes" Oct 30 21:54:59.581583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:59.581537 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" podUID="a79f6755-8dab-4c54-b8e1-db5a5e032726" containerName="payload-processing" containerID="cri-o://cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d" gracePeriod=25 Oct 30 21:54:59.680033 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:54:59.679981 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" podUID="da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" containerName="payload-pre-processing" containerID="cri-o://8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b" gracePeriod=25 Oct 30 21:55:02.601076 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:02.600974 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" containerID="cri-o://0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023" gracePeriod=45 Oct 30 21:55:02.623420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:02.623360 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" containerID="cri-o://03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e" gracePeriod=45 Oct 30 21:55:02.811955 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:02.811912 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused" Oct 30 21:55:02.923508 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:02.923470 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:55:02.926925 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:02.926899 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:55:03.039839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.039803 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec879707-8027-4f91-8faf-a9b97e4a1267-kube-api-access-nzt85\" (UniqueName: \"kubernetes.io/projected/ec879707-8027-4f91-8faf-a9b97e4a1267-kube-api-access-nzt85\") pod \"ec879707-8027-4f91-8faf-a9b97e4a1267\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " Oct 30 21:55:03.039839 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.039853 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-dshm\") pod \"ec879707-8027-4f91-8faf-a9b97e4a1267\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " Oct 30 21:55:03.040126 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.039887 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-tmp-dir\") pod \"ea8af78e-f20e-4709-8198-3d175d15bba3\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " Oct 30 21:55:03.040126 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.039941 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-model-cache\") pod \"ea8af78e-f20e-4709-8198-3d175d15bba3\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " Oct 30 21:55:03.040126 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040077 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8af78e-f20e-4709-8198-3d175d15bba3-kube-api-access-kzpv4\" (UniqueName: \"kubernetes.io/projected/ea8af78e-f20e-4709-8198-3d175d15bba3-kube-api-access-kzpv4\") pod \"ea8af78e-f20e-4709-8198-3d175d15bba3\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " Oct 30 21:55:03.040308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040134 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-home\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-home\") pod \"ec879707-8027-4f91-8faf-a9b97e4a1267\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " Oct 30 21:55:03.040308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040171 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea8af78e-f20e-4709-8198-3d175d15bba3-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8af78e-f20e-4709-8198-3d175d15bba3-tls-certs\") pod \"ea8af78e-f20e-4709-8198-3d175d15bba3\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " Oct 30 21:55:03.040308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040200 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-tmp-dir\") pod \"ec879707-8027-4f91-8faf-a9b97e4a1267\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " Oct 30 21:55:03.040308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040281 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-model-cache\") pod \"ec879707-8027-4f91-8faf-a9b97e4a1267\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " Oct 30 21:55:03.040566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040335 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec879707-8027-4f91-8faf-a9b97e4a1267-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec879707-8027-4f91-8faf-a9b97e4a1267-tls-certs\") pod \"ec879707-8027-4f91-8faf-a9b97e4a1267\" (UID: \"ec879707-8027-4f91-8faf-a9b97e4a1267\") " Oct 30 21:55:03.040566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040365 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-home\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-home\") pod \"ea8af78e-f20e-4709-8198-3d175d15bba3\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " Oct 30 21:55:03.040566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040394 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-dshm\") pod \"ea8af78e-f20e-4709-8198-3d175d15bba3\" (UID: \"ea8af78e-f20e-4709-8198-3d175d15bba3\") " Oct 30 21:55:03.040566 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.040436 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-model-cache" pod "ea8af78e-f20e-4709-8198-3d175d15bba3" (UID: "ea8af78e-f20e-4709-8198-3d175d15bba3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:03.041340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.041050 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-tmp-dir" pod "ec879707-8027-4f91-8faf-a9b97e4a1267" (UID: "ec879707-8027-4f91-8faf-a9b97e4a1267"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:03.041340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.041127 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.044032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.043998 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec879707-8027-4f91-8faf-a9b97e4a1267-kube-api-access-nzt85" pod "ec879707-8027-4f91-8faf-a9b97e4a1267" (UID: "ec879707-8027-4f91-8faf-a9b97e4a1267"). InnerVolumeSpecName "kube-api-access-nzt85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:03.044226 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.044202 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-tmp-dir" pod "ea8af78e-f20e-4709-8198-3d175d15bba3" (UID: "ea8af78e-f20e-4709-8198-3d175d15bba3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:03.044326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.044222 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-model-cache" pod "ec879707-8027-4f91-8faf-a9b97e4a1267" (UID: "ec879707-8027-4f91-8faf-a9b97e4a1267"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:03.044417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.044151 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-home" pod "ea8af78e-f20e-4709-8198-3d175d15bba3" (UID: "ea8af78e-f20e-4709-8198-3d175d15bba3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:03.044545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.044414 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-home" pod "ec879707-8027-4f91-8faf-a9b97e4a1267" (UID: "ec879707-8027-4f91-8faf-a9b97e4a1267"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:03.044882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.044824 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-dshm" pod "ea8af78e-f20e-4709-8198-3d175d15bba3" (UID: "ea8af78e-f20e-4709-8198-3d175d15bba3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:03.045085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.045064 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8af78e-f20e-4709-8198-3d175d15bba3-kube-api-access-kzpv4" pod "ea8af78e-f20e-4709-8198-3d175d15bba3" (UID: "ea8af78e-f20e-4709-8198-3d175d15bba3"). InnerVolumeSpecName "kube-api-access-kzpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:03.045598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.045573 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-dshm" pod "ec879707-8027-4f91-8faf-a9b97e4a1267" (UID: "ec879707-8027-4f91-8faf-a9b97e4a1267"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:03.046259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.046233 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec879707-8027-4f91-8faf-a9b97e4a1267-tls-certs" pod "ec879707-8027-4f91-8faf-a9b97e4a1267" (UID: "ec879707-8027-4f91-8faf-a9b97e4a1267"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:03.046455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.046436 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8af78e-f20e-4709-8198-3d175d15bba3-tls-certs" pod "ea8af78e-f20e-4709-8198-3d175d15bba3" (UID: "ea8af78e-f20e-4709-8198-3d175d15bba3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142234 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzpv4\" (UniqueName: \"kubernetes.io/projected/ea8af78e-f20e-4709-8198-3d175d15bba3-kube-api-access-kzpv4\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142268 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142279 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea8af78e-f20e-4709-8198-3d175d15bba3-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142287 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142295 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142303 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec879707-8027-4f91-8faf-a9b97e4a1267-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142311 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142320 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142328 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzt85\" (UniqueName: \"kubernetes.io/projected/ec879707-8027-4f91-8faf-a9b97e4a1267-kube-api-access-nzt85\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142336 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec879707-8027-4f91-8faf-a9b97e4a1267-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.142810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.142344 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea8af78e-f20e-4709-8198-3d175d15bba3-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:03.239344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.239314 4526 generic.go:356] "Generic (PLEG): container finished" podID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerID="0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023" exitCode=0 Oct 30 21:55:03.239583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.239361 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" Oct 30 21:55:03.239583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.239397 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" event={"ID":"ec879707-8027-4f91-8faf-a9b97e4a1267","Type":"ContainerDied","Data":"0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023"} Oct 30 21:55:03.239583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.239445 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" event={"ID":"ec879707-8027-4f91-8faf-a9b97e4a1267","Type":"ContainerDied","Data":"0799686922a30e8a3cc59529368ee310210959bec50dbde614f4246a6e12b3b1"} Oct 30 21:55:03.239583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.239467 4526 scope.go:122] "RemoveContainer" containerID="0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023" Oct 30 21:55:03.241071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.241046 4526 generic.go:356] "Generic (PLEG): container finished" podID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerID="03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e" exitCode=0 Oct 30 21:55:03.241195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.241108 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" Oct 30 21:55:03.241195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.241121 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" event={"ID":"ea8af78e-f20e-4709-8198-3d175d15bba3","Type":"ContainerDied","Data":"03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e"} Oct 30 21:55:03.241195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.241169 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5" event={"ID":"ea8af78e-f20e-4709-8198-3d175d15bba3","Type":"ContainerDied","Data":"e9e818cdce2f84f58e32be661291daf11720428f5547506b6359cddfbfbb17ef"} Oct 30 21:55:03.251970 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.251946 4526 scope.go:122] "RemoveContainer" containerID="0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023" Oct 30 21:55:03.252375 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:03.252354 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023\": container with ID starting with 0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023 not found: ID does not exist" containerID="0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023" Oct 30 21:55:03.252450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.252390 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023"} err="failed to get container status \"0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023\": rpc error: code = NotFound desc = could not find container \"0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023\": container with ID starting with 0186df8e19f22ed99745702adbd50efc4d2ae4f850232581f0d059ed542fc023 not found: ID does not exist" Oct 30 21:55:03.252450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.252413 4526 scope.go:122] "RemoveContainer" containerID="03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e" Oct 30 21:55:03.263453 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.263429 4526 scope.go:122] "RemoveContainer" containerID="03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e" Oct 30 21:55:03.263942 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:03.263903 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e\": container with ID starting with 03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e not found: ID does not exist" containerID="03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e" Oct 30 21:55:03.264074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.263941 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e"} err="failed to get container status \"03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e\": rpc error: code = NotFound desc = could not find container \"03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e\": container with ID starting with 03b18ad3a74909beb105d4ce74eadab63a81cffa9af4aca6956a57402e4afe8e not found: ID does not exist" Oct 30 21:55:03.413691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.413596 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5"] Oct 30 21:55:03.419218 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.419183 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/simulator-w3-9796f1-kserve-b8bcdcc7b-f7wx5"] Oct 30 21:55:03.531548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.531511 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4"] Oct 30 21:55:03.535379 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.535346 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4"] Oct 30 21:55:03.765470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.765334 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm"] Oct 30 21:55:03.768799 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.768764 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fbb72e-62jzm"] Oct 30 21:55:03.826779 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.826741 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9796f1/distinct-w3-9796f1-kserve-6cdd7dbffb-tpgh4" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:03.986038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.986003 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77cca1b4-de1d-4ae1-91e8-032e7b9b24ed" path="/var/lib/kubelet/pods/77cca1b4-de1d-4ae1-91e8-032e7b9b24ed/volumes" Oct 30 21:55:03.986467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.986454 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" path="/var/lib/kubelet/pods/ea8af78e-f20e-4709-8198-3d175d15bba3/volumes" Oct 30 21:55:03.986868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.986856 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" path="/var/lib/kubelet/pods/ec879707-8027-4f91-8faf-a9b97e4a1267/volumes" Oct 30 21:55:03.995741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:03.995711 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" podUID="da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:55:05.000700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:05.000664 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" podUID="a79f6755-8dab-4c54-b8e1-db5a5e032726" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:55:08.549564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.549533 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5"] Oct 30 21:55:08.550128 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550113 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" Oct 30 21:55:08.550174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550129 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" Oct 30 21:55:08.550174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550160 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" Oct 30 21:55:08.550174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550172 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" Oct 30 21:55:08.550265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550183 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" Oct 30 21:55:08.550265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550188 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" Oct 30 21:55:08.550265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550199 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" Oct 30 21:55:08.550265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550204 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" Oct 30 21:55:08.550265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550216 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3bf32042-919c-4a6d-b379-676feb9106b7" containerName="revoke-keys" Oct 30 21:55:08.550265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550221 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf32042-919c-4a6d-b379-676feb9106b7" containerName="revoke-keys" Oct 30 21:55:08.550265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550230 4526 cpu_manager.go:414] "containerMap: removing container" podUID="05f408a1-4cc4-455d-ae9b-07b577205888" containerName="revoke-keys" Oct 30 21:55:08.550265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550235 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f408a1-4cc4-455d-ae9b-07b577205888" containerName="revoke-keys" Oct 30 21:55:08.550532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550320 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8af78e-f20e-4709-8198-3d175d15bba3" containerName="main" Oct 30 21:55:08.550532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550329 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec879707-8027-4f91-8faf-a9b97e4a1267" containerName="main" Oct 30 21:55:08.550532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550336 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" Oct 30 21:55:08.550532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550342 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf32042-919c-4a6d-b379-676feb9106b7" containerName="revoke-keys" Oct 30 21:55:08.550532 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.550348 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f408a1-4cc4-455d-ae9b-07b577205888" containerName="revoke-keys" Oct 30 21:55:08.554937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.554913 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.628515 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.627546 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5"] Oct 30 21:55:08.665675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.665640 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk7w4\"" Oct 30 21:55:08.665873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.665763 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:08.665873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.665856 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:08.665959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.665907 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e9aa8e-serving-cert\"" Oct 30 21:55:08.666019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.666003 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:55:08.726180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.726145 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.726335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.726194 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97q62\" (UniqueName: \"kubernetes.io/projected/37b158e4-91ec-4c81-8223-e8394d7deb1b-kube-api-access-97q62\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.726335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.726234 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b158e4-91ec-4c81-8223-e8394d7deb1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.726335 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.726256 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-metrics-certs\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.828088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.827991 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.828088 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.828040 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q62\" (UniqueName: \"kubernetes.io/projected/37b158e4-91ec-4c81-8223-e8394d7deb1b-kube-api-access-97q62\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.828329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.828097 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b158e4-91ec-4c81-8223-e8394d7deb1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.828329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.828120 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-metrics-certs\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.830276 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.830246 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b158e4-91ec-4c81-8223-e8394d7deb1b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.834783 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.834000 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-metrics-certs\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.837136 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.837106 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:08.892511 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:08.892467 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q62\" (UniqueName: \"kubernetes.io/projected/37b158e4-91ec-4c81-8223-e8394d7deb1b-kube-api-access-97q62\") pod \"maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:09.174565 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.173525 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:09.344867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.344828 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7"] Oct 30 21:55:09.345563 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.345547 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dc33b1-e449-4b7f-b1eb-10adff633529" containerName="istio-proxy" Oct 30 21:55:09.349887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.349866 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.403182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.403147 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:09.404154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.403267 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:55:09.407194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.407165 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7"] Oct 30 21:55:09.440312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.440208 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.440312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.440266 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgm7\" (UniqueName: \"kubernetes.io/projected/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-kube-api-access-gdgm7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.440614 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.440480 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.542408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.542367 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.542647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.542443 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgm7\" (UniqueName: \"kubernetes.io/projected/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-kube-api-access-gdgm7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.542647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.542583 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.543114 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.543091 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.545194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.545169 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.600239 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.600204 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgm7\" (UniqueName: \"kubernetes.io/projected/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-kube-api-access-gdgm7\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:09.632186 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.632156 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5"] Oct 30 21:55:09.633394 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:55:09.633369 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b158e4_91ec_4c81_8223_e8394d7deb1b.slice/crio-652f3d1b7750d50efd3dc12a61f266244492321a36bd31ff0af5fb29b1150a12 WatchSource:0}: Error finding container 652f3d1b7750d50efd3dc12a61f266244492321a36bd31ff0af5fb29b1150a12: Status 404 returned error can't find the container with id 652f3d1b7750d50efd3dc12a61f266244492321a36bd31ff0af5fb29b1150a12 Oct 30 21:55:09.634797 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.634778 4526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:55:09.665657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:09.665631 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:10.043717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:10.043689 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7"] Oct 30 21:55:10.044675 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:55:10.044652 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a13c39_65d4_4e4e_8d8e_fa90e603edc0.slice/crio-be4f2e9464dc69338078596e2cf839986232c4a3404b1263655013bba460aa7b WatchSource:0}: Error finding container be4f2e9464dc69338078596e2cf839986232c4a3404b1263655013bba460aa7b: Status 404 returned error can't find the container with id be4f2e9464dc69338078596e2cf839986232c4a3404b1263655013bba460aa7b Oct 30 21:55:10.293617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:10.293522 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" event={"ID":"37b158e4-91ec-4c81-8223-e8394d7deb1b","Type":"ContainerStarted","Data":"652f3d1b7750d50efd3dc12a61f266244492321a36bd31ff0af5fb29b1150a12"} Oct 30 21:55:10.294971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:10.294944 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" event={"ID":"38a13c39-65d4-4e4e-8d8e-fa90e603edc0","Type":"ContainerStarted","Data":"be4f2e9464dc69338078596e2cf839986232c4a3404b1263655013bba460aa7b"} Oct 30 21:55:11.304721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:11.304688 4526 generic.go:356] "Generic (PLEG): container finished" podID="38a13c39-65d4-4e4e-8d8e-fa90e603edc0" containerID="39487ea0faf40917202fc2ac4f5518e96b35492bdf937be8e761eaef94d87b79" exitCode=0 Oct 30 21:55:11.305263 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:11.304772 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" event={"ID":"38a13c39-65d4-4e4e-8d8e-fa90e603edc0","Type":"ContainerDied","Data":"39487ea0faf40917202fc2ac4f5518e96b35492bdf937be8e761eaef94d87b79"} Oct 30 21:55:12.087378 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.087343 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh"] Oct 30 21:55:12.091365 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.091346 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.154400 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.154355 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh"] Oct 30 21:55:12.174080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.174045 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xsp7\" (UniqueName: \"kubernetes.io/projected/db567ebe-3deb-409b-b7ed-737fbd5a1518-kube-api-access-8xsp7\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.174223 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.174108 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.174308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.174282 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.275887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.275831 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xsp7\" (UniqueName: \"kubernetes.io/projected/db567ebe-3deb-409b-b7ed-737fbd5a1518-kube-api-access-8xsp7\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.276080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.275933 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.276080 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.276007 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.276739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.276715 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.278523 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.278504 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.313688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.313656 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" event={"ID":"37b158e4-91ec-4c81-8223-e8394d7deb1b","Type":"ContainerStarted","Data":"6588e3cc9c03c8ac05d379e7ff4d9289b6aea4dee90d740a39aece1425272c8b"} Oct 30 21:55:12.314115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.313965 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:12.334585 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.334541 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xsp7\" (UniqueName: \"kubernetes.io/projected/db567ebe-3deb-409b-b7ed-737fbd5a1518-kube-api-access-8xsp7\") pod \"maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.404849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.404816 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:12.442746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.442687 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" podStartSLOduration=2.48091268 podStartE2EDuration="4.442663009s" totalImagesPullingTime="1.961750329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:12.437609711 +0000 UTC m=+2302.570982912" watchObservedRunningTime="2026-10-30 21:55:12.442663009 +0000 UTC m=+2302.576036207" Oct 30 21:55:12.482141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.482112 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:12.495387 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:12.495353 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:55:12.587951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.587916 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-xks-ca\") pod \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " Oct 30 21:55:12.588166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.588014 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-kube-api-access-gdgm7\" (UniqueName: \"kubernetes.io/projected/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-kube-api-access-gdgm7\") pod \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " Oct 30 21:55:12.588166 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.588140 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-openshift-ca\") pod \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\" (UID: \"38a13c39-65d4-4e4e-8d8e-fa90e603edc0\") " Oct 30 21:55:12.588656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.588618 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-openshift-ca" pod "38a13c39-65d4-4e4e-8d8e-fa90e603edc0" (UID: "38a13c39-65d4-4e4e-8d8e-fa90e603edc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:12.588950 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.588742 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:12.590222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.590194 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-xks-ca" pod "38a13c39-65d4-4e4e-8d8e-fa90e603edc0" (UID: "38a13c39-65d4-4e4e-8d8e-fa90e603edc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:12.590357 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.590334 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-kube-api-access-gdgm7" pod "38a13c39-65d4-4e4e-8d8e-fa90e603edc0" (UID: "38a13c39-65d4-4e4e-8d8e-fa90e603edc0"). InnerVolumeSpecName "kube-api-access-gdgm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:12.689937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.689864 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:12.689937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.689894 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdgm7\" (UniqueName: \"kubernetes.io/projected/38a13c39-65d4-4e4e-8d8e-fa90e603edc0-kube-api-access-gdgm7\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:12.804426 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:12.804401 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh"] Oct 30 21:55:12.806080 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:55:12.806056 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb567ebe_3deb_409b_b7ed_737fbd5a1518.slice/crio-e38fa3ce39566d568cbce56abe37f93c268a9eb301d217da077bb33b23cde9ee WatchSource:0}: Error finding container e38fa3ce39566d568cbce56abe37f93c268a9eb301d217da077bb33b23cde9ee: Status 404 returned error can't find the container with id e38fa3ce39566d568cbce56abe37f93c268a9eb301d217da077bb33b23cde9ee Oct 30 21:55:13.222159 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.222127 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh"] Oct 30 21:55:13.222345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.222321 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" containerID="cri-o://56de2d119a76d3088223439b36b044a77006e86db1d5131e6df909e920fb8dd4" gracePeriod=30 Oct 30 21:55:13.323127 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.323094 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" event={"ID":"db567ebe-3deb-409b-b7ed-737fbd5a1518","Type":"ContainerStarted","Data":"62fc7951f857da1f7ae03fa9eddefc645dc5fd677bd7a4620514231f160ed28a"} Oct 30 21:55:13.323617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.323162 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" event={"ID":"db567ebe-3deb-409b-b7ed-737fbd5a1518","Type":"ContainerStarted","Data":"e38fa3ce39566d568cbce56abe37f93c268a9eb301d217da077bb33b23cde9ee"} Oct 30 21:55:13.325360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.325326 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" event={"ID":"38a13c39-65d4-4e4e-8d8e-fa90e603edc0","Type":"ContainerDied","Data":"be4f2e9464dc69338078596e2cf839986232c4a3404b1263655013bba460aa7b"} Oct 30 21:55:13.325360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.325350 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7" Oct 30 21:55:13.325578 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.325384 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be4f2e9464dc69338078596e2cf839986232c4a3404b1263655013bba460aa7b" Oct 30 21:55:13.522665 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.522614 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" podStartSLOduration=1.522600008 podStartE2EDuration="1.522600008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:13.519330516 +0000 UTC m=+2303.652703715" watchObservedRunningTime="2026-10-30 21:55:13.522600008 +0000 UTC m=+2303.655973206" Oct 30 21:55:13.996449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:13.996406 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" podUID="da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:55:14.333702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:14.333667 4526 generic.go:356] "Generic (PLEG): container finished" podID="db567ebe-3deb-409b-b7ed-737fbd5a1518" containerID="62fc7951f857da1f7ae03fa9eddefc645dc5fd677bd7a4620514231f160ed28a" exitCode=0 Oct 30 21:55:14.334073 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:14.333725 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" event={"ID":"db567ebe-3deb-409b-b7ed-737fbd5a1518","Type":"ContainerDied","Data":"62fc7951f857da1f7ae03fa9eddefc645dc5fd677bd7a4620514231f160ed28a"} Oct 30 21:55:15.000772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.000735 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" podUID="a79f6755-8dab-4c54-b8e1-db5a5e032726" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:55:15.491606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.491579 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:15.525160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.525125 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-xks-ca\") pod \"db567ebe-3deb-409b-b7ed-737fbd5a1518\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " Oct 30 21:55:15.525346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.525264 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-openshift-ca\") pod \"db567ebe-3deb-409b-b7ed-737fbd5a1518\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " Oct 30 21:55:15.525346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.525291 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db567ebe-3deb-409b-b7ed-737fbd5a1518-kube-api-access-8xsp7\" (UniqueName: \"kubernetes.io/projected/db567ebe-3deb-409b-b7ed-737fbd5a1518-kube-api-access-8xsp7\") pod \"db567ebe-3deb-409b-b7ed-737fbd5a1518\" (UID: \"db567ebe-3deb-409b-b7ed-737fbd5a1518\") " Oct 30 21:55:15.525724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.525696 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-openshift-ca" pod "db567ebe-3deb-409b-b7ed-737fbd5a1518" (UID: "db567ebe-3deb-409b-b7ed-737fbd5a1518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:15.526017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.525998 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:15.527421 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.527402 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-xks-ca" pod "db567ebe-3deb-409b-b7ed-737fbd5a1518" (UID: "db567ebe-3deb-409b-b7ed-737fbd5a1518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:15.527586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.527565 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db567ebe-3deb-409b-b7ed-737fbd5a1518-kube-api-access-8xsp7" pod "db567ebe-3deb-409b-b7ed-737fbd5a1518" (UID: "db567ebe-3deb-409b-b7ed-737fbd5a1518"). InnerVolumeSpecName "kube-api-access-8xsp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:15.628007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.627922 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db567ebe-3deb-409b-b7ed-737fbd5a1518-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:15.628007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:15.627953 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xsp7\" (UniqueName: \"kubernetes.io/projected/db567ebe-3deb-409b-b7ed-737fbd5a1518-kube-api-access-8xsp7\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:16.348123 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:16.348093 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" Oct 30 21:55:16.348334 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:16.348127 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh" event={"ID":"db567ebe-3deb-409b-b7ed-737fbd5a1518","Type":"ContainerDied","Data":"e38fa3ce39566d568cbce56abe37f93c268a9eb301d217da077bb33b23cde9ee"} Oct 30 21:55:16.348334 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:16.348175 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e38fa3ce39566d568cbce56abe37f93c268a9eb301d217da077bb33b23cde9ee" Oct 30 21:55:17.092777 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:17.092740 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-pg8wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:55:17.093154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:17.092791 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:55:18.332992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.332959 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 21:55:18.364584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.364550 4526 generic.go:356] "Generic (PLEG): container finished" podID="00307618-1d57-4089-aaf6-5b1436741bcf" containerID="56de2d119a76d3088223439b36b044a77006e86db1d5131e6df909e920fb8dd4" exitCode=0 Oct 30 21:55:18.364584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.364584 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" event={"ID":"00307618-1d57-4089-aaf6-5b1436741bcf","Type":"ContainerDied","Data":"56de2d119a76d3088223439b36b044a77006e86db1d5131e6df909e920fb8dd4"} Oct 30 21:55:18.498188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.498113 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:55:18.565255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565225 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-data\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565444 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565305 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-istio-token\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-istio-token\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565444 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565343 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00307618-1d57-4089-aaf6-5b1436741bcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00307618-1d57-4089-aaf6-5b1436741bcf-istiod-ca-cert\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565444 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565371 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-credential-socket\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565447 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-socket\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565548 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00307618-1d57-4089-aaf6-5b1436741bcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00307618-1d57-4089-aaf6-5b1436741bcf-istio-podinfo\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565571 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-data" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.565621 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565582 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-certs\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565670 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-envoy\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565719 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-kube-api-access-hdmw8\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-kube-api-access-hdmw8\") pod \"00307618-1d57-4089-aaf6-5b1436741bcf\" (UID: \"00307618-1d57-4089-aaf6-5b1436741bcf\") " Oct 30 21:55:18.565848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565801 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-certs" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.565848 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.565818 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00307618-1d57-4089-aaf6-5b1436741bcf-istiod-ca-cert" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:18.566253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.566223 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.566253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.566243 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00307618-1d57-4089-aaf6-5b1436741bcf-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.566253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.566256 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.566679 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.566265 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-socket" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.566679 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.566324 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-credential-socket" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.567827 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.567804 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-istio-token" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.567954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.567939 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00307618-1d57-4089-aaf6-5b1436741bcf-istio-podinfo" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:18.568716 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.568688 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-envoy" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.568797 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.568732 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-kube-api-access-hdmw8" pod "00307618-1d57-4089-aaf6-5b1436741bcf" (UID: "00307618-1d57-4089-aaf6-5b1436741bcf"). InnerVolumeSpecName "kube-api-access-hdmw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.667936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.667899 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdmw8\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-kube-api-access-hdmw8\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.667936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.667932 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00307618-1d57-4089-aaf6-5b1436741bcf-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.667936 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.667945 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.668198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.667957 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.668198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.667971 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00307618-1d57-4089-aaf6-5b1436741bcf-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.668198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.667982 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00307618-1d57-4089-aaf6-5b1436741bcf-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.998178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.998136 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm"] Oct 30 21:55:18.998414 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:18.998371 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" containerID="cri-o://80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a" gracePeriod=30 Oct 30 21:55:19.372304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.372272 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" Oct 30 21:55:19.372304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.372284 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh" event={"ID":"00307618-1d57-4089-aaf6-5b1436741bcf","Type":"ContainerDied","Data":"39c679eda318231d43a80b5552367119f915717f2dd6f9eecd9ab9b3a80b11ce"} Oct 30 21:55:19.372929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.372388 4526 scope.go:122] "RemoveContainer" containerID="56de2d119a76d3088223439b36b044a77006e86db1d5131e6df909e920fb8dd4" Oct 30 21:55:19.553926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.553890 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh"] Oct 30 21:55:19.559245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.559205 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-pg8wh"] Oct 30 21:55:19.876326 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.873926 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:55:19.952833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.952806 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:55:19.986155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.986120 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" path="/var/lib/kubelet/pods/00307618-1d57-4089-aaf6-5b1436741bcf/volumes" Oct 30 21:55:19.989968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.989933 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79f6755-8dab-4c54-b8e1-db5a5e032726-kube-api-access-26jpc\" (UniqueName: \"kubernetes.io/projected/a79f6755-8dab-4c54-b8e1-db5a5e032726-kube-api-access-26jpc\") pod \"a79f6755-8dab-4c54-b8e1-db5a5e032726\" (UID: \"a79f6755-8dab-4c54-b8e1-db5a5e032726\") " Oct 30 21:55:19.990133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.989997 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79f6755-8dab-4c54-b8e1-db5a5e032726-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79f6755-8dab-4c54-b8e1-db5a5e032726-plugins-config-volume\") pod \"a79f6755-8dab-4c54-b8e1-db5a5e032726\" (UID: \"a79f6755-8dab-4c54-b8e1-db5a5e032726\") " Oct 30 21:55:19.990341 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.990321 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79f6755-8dab-4c54-b8e1-db5a5e032726-plugins-config-volume" pod "a79f6755-8dab-4c54-b8e1-db5a5e032726" (UID: "a79f6755-8dab-4c54-b8e1-db5a5e032726"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:19.992187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:19.992165 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79f6755-8dab-4c54-b8e1-db5a5e032726-kube-api-access-26jpc" pod "a79f6755-8dab-4c54-b8e1-db5a5e032726" (UID: "a79f6755-8dab-4c54-b8e1-db5a5e032726"). InnerVolumeSpecName "kube-api-access-26jpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:20.090941 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.090898 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-plugins-config-volume\") pod \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\" (UID: \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\") " Oct 30 21:55:20.091169 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.091156 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-kube-api-access-6pq92\" (UniqueName: \"kubernetes.io/projected/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-kube-api-access-6pq92\") pod \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\" (UID: \"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85\") " Oct 30 21:55:20.091278 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.091254 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-plugins-config-volume" pod "da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" (UID: "da0a50aa-2cca-40ab-8ca0-794f5b0dcf85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:20.091776 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.091746 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:20.091919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.091780 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26jpc\" (UniqueName: \"kubernetes.io/projected/a79f6755-8dab-4c54-b8e1-db5a5e032726-kube-api-access-26jpc\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:20.091919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.091795 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a79f6755-8dab-4c54-b8e1-db5a5e032726-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:20.093688 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.093665 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-kube-api-access-6pq92" pod "da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" (UID: "da0a50aa-2cca-40ab-8ca0-794f5b0dcf85"). InnerVolumeSpecName "kube-api-access-6pq92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:20.192807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.192722 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pq92\" (UniqueName: \"kubernetes.io/projected/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85-kube-api-access-6pq92\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:20.380505 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.380465 4526 generic.go:356] "Generic (PLEG): container finished" podID="a79f6755-8dab-4c54-b8e1-db5a5e032726" containerID="cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d" exitCode=0 Oct 30 21:55:20.380981 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.380529 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" Oct 30 21:55:20.380981 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.380547 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" event={"ID":"a79f6755-8dab-4c54-b8e1-db5a5e032726","Type":"ContainerDied","Data":"cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d"} Oct 30 21:55:20.380981 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.380604 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g" event={"ID":"a79f6755-8dab-4c54-b8e1-db5a5e032726","Type":"ContainerDied","Data":"16fc34d4e4cfe68a648bd926d83d4e922c3e956f4961ba4f0de88d1140d87a9b"} Oct 30 21:55:20.380981 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.380633 4526 scope.go:122] "RemoveContainer" containerID="cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d" Oct 30 21:55:20.382410 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.382387 4526 generic.go:356] "Generic (PLEG): container finished" podID="da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" containerID="8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b" exitCode=0 Oct 30 21:55:20.382659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.382428 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" Oct 30 21:55:20.382659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.382523 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" event={"ID":"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85","Type":"ContainerDied","Data":"8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b"} Oct 30 21:55:20.382659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.382582 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv" event={"ID":"da0a50aa-2cca-40ab-8ca0-794f5b0dcf85","Type":"ContainerDied","Data":"8d87b98405c1c0d70eb82033c2a34fc34383623090d4f78d9a1cf6acda1dfc07"} Oct 30 21:55:20.392894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.392867 4526 scope.go:122] "RemoveContainer" containerID="cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d" Oct 30 21:55:20.393179 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:20.393155 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d\": container with ID starting with cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d not found: ID does not exist" containerID="cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d" Oct 30 21:55:20.393251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.393187 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d"} err="failed to get container status \"cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d\": rpc error: code = NotFound desc = could not find container \"cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d\": container with ID starting with cc66d3cca8ed1a1ceaca5ee8d49f22a67630ba484f8332dfd3cf094186060c2d not found: ID does not exist" Oct 30 21:55:20.393251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.393206 4526 scope.go:122] "RemoveContainer" containerID="8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b" Oct 30 21:55:20.403741 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.403719 4526 scope.go:122] "RemoveContainer" containerID="8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b" Oct 30 21:55:20.404073 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:20.404055 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b\": container with ID starting with 8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b not found: ID does not exist" containerID="8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b" Oct 30 21:55:20.404131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.404081 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b"} err="failed to get container status \"8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b\": rpc error: code = NotFound desc = could not find container \"8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b\": container with ID starting with 8b4f26585ce680e30196fed630517340c948486567759826b4b1dfe46bdb899b not found: ID does not exist" Oct 30 21:55:20.545775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.545744 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g"] Oct 30 21:55:20.594954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.594920 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9796f1-6799974f65-tjn5g"] Oct 30 21:55:20.708649 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.708613 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv"] Oct 30 21:55:20.713061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.713035 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9796f1-fc64697f-x2phv"] Oct 30 21:55:20.870225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:20.870137 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt"] Oct 30 21:55:21.104807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.104771 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c"] Oct 30 21:55:21.105351 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105336 4526 cpu_manager.go:414] "containerMap: removing container" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" Oct 30 21:55:21.105404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105353 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" Oct 30 21:55:21.105404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105373 4526 cpu_manager.go:414] "containerMap: removing container" podUID="da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" containerName="payload-pre-processing" Oct 30 21:55:21.105404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105380 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" containerName="payload-pre-processing" Oct 30 21:55:21.105545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105411 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a79f6755-8dab-4c54-b8e1-db5a5e032726" containerName="payload-processing" Oct 30 21:55:21.105545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105417 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79f6755-8dab-4c54-b8e1-db5a5e032726" containerName="payload-processing" Oct 30 21:55:21.105545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105430 4526 cpu_manager.go:414] "containerMap: removing container" podUID="db567ebe-3deb-409b-b7ed-737fbd5a1518" containerName="revoke-keys" Oct 30 21:55:21.105545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105435 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db567ebe-3deb-409b-b7ed-737fbd5a1518" containerName="revoke-keys" Oct 30 21:55:21.105545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105453 4526 cpu_manager.go:414] "containerMap: removing container" podUID="38a13c39-65d4-4e4e-8d8e-fa90e603edc0" containerName="revoke-keys" Oct 30 21:55:21.105545 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105459 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a13c39-65d4-4e4e-8d8e-fa90e603edc0" containerName="revoke-keys" Oct 30 21:55:21.106129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105552 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a13c39-65d4-4e4e-8d8e-fa90e603edc0" containerName="revoke-keys" Oct 30 21:55:21.106129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105562 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="00307618-1d57-4089-aaf6-5b1436741bcf" containerName="istio-proxy" Oct 30 21:55:21.106129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105570 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" containerName="payload-pre-processing" Oct 30 21:55:21.106129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105580 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79f6755-8dab-4c54-b8e1-db5a5e032726" containerName="payload-processing" Oct 30 21:55:21.106129 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.105587 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="db567ebe-3deb-409b-b7ed-737fbd5a1518" containerName="revoke-keys" Oct 30 21:55:21.110857 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.110836 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.172053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.171950 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c"] Oct 30 21:55:21.215321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215279 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.215321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215316 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcqp\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-kube-api-access-5xcqp\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.215631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215337 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.215631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215454 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.215631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215570 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.215816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215693 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.215816 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215798 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.215934 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215886 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.215993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.215952 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb6a42-f6f9-47be-a5cc-ede068832de1-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.221916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.221891 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-75c2acf6-openshift-default-dockercfg-q8244\"" Oct 30 21:55:21.317675 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.317640 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.317891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.317722 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.317891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.317817 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.317891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.317880 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.318065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.317952 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.318065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.317985 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb6a42-f6f9-47be-a5cc-ede068832de1-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.318065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.318060 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.318224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.318092 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcqp\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-kube-api-access-5xcqp\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.318224 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.318128 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.318757 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.318730 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.318937 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.318912 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.319040 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.319017 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.319190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.319167 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.319277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.319191 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb6a42-f6f9-47be-a5cc-ede068832de1-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.321184 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.321155 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.321586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.321566 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.376443 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.376405 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.376666 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.376603 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcqp\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-kube-api-access-5xcqp\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.433805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.433704 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:21.887113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.887085 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c"] Oct 30 21:55:21.887961 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:55:21.887932 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bb6a42_f6f9_47be_a5cc_ede068832de1.slice/crio-e9e12b72343203dd93ec0f5c1b9b13352f65585293bf85fb56faeb900e70dabc WatchSource:0}: Error finding container e9e12b72343203dd93ec0f5c1b9b13352f65585293bf85fb56faeb900e70dabc: Status 404 returned error can't find the container with id e9e12b72343203dd93ec0f5c1b9b13352f65585293bf85fb56faeb900e70dabc Oct 30 21:55:21.890432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.890400 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:21.890568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.890467 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:21.890568 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.890540 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:21.986723 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.986678 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79f6755-8dab-4c54-b8e1-db5a5e032726" path="/var/lib/kubelet/pods/a79f6755-8dab-4c54-b8e1-db5a5e032726/volumes" Oct 30 21:55:21.987322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:21.987300 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0a50aa-2cca-40ab-8ca0-794f5b0dcf85" path="/var/lib/kubelet/pods/da0a50aa-2cca-40ab-8ca0-794f5b0dcf85/volumes" Oct 30 21:55:22.405992 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:22.405951 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" event={"ID":"90bb6a42-f6f9-47be-a5cc-ede068832de1","Type":"ContainerStarted","Data":"9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d"} Oct 30 21:55:22.406214 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:22.406023 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" event={"ID":"90bb6a42-f6f9-47be-a5cc-ede068832de1","Type":"ContainerStarted","Data":"e9e12b72343203dd93ec0f5c1b9b13352f65585293bf85fb56faeb900e70dabc"} Oct 30 21:55:22.434396 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:22.434355 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:22.441235 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:22.441203 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:22.528476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:22.528408 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" podStartSLOduration=1.528378477 podStartE2EDuration="1.528378477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:22.523323779 +0000 UTC m=+2312.656696978" watchObservedRunningTime="2026-10-30 21:55:22.528378477 +0000 UTC m=+2312.661751675" Oct 30 21:55:23.412891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:23.412862 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:23.413945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:23.413923 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:55:24.295441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.295419 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:55:24.384320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384284 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-credential-socket\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.384320 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384320 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-envoy\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.384630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384394 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-token\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-token\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.384630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384569 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istiod-ca-cert\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.384630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384604 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-certs\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.384792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384664 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-podinfo\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.384792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384702 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-credential-socket" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:24.384792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384751 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-data\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.384792 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384783 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-kube-api-access-q2748\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-kube-api-access-q2748\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.385013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384820 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-socket\") pod \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\" (UID: \"c67bb07b-9ee6-412a-a850-c872cb33b1b2\") " Oct 30 21:55:24.385013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384927 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-certs" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:24.385013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.384998 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istiod-ca-cert" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:24.385251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.385226 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-socket" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:24.385251 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.385241 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-data" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:24.385742 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.385717 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.385852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.385746 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.385852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.385762 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.385852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.385776 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.385852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.385791 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.387093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.387063 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-envoy" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:24.387212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.387094 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-podinfo" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:24.387212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.387146 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-token" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:24.387212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.387177 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-kube-api-access-q2748" pod "c67bb07b-9ee6-412a-a850-c872cb33b1b2" (UID: "c67bb07b-9ee6-412a-a850-c872cb33b1b2"). InnerVolumeSpecName "kube-api-access-q2748". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:24.421677 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.421645 4526 generic.go:356] "Generic (PLEG): container finished" podID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerID="80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a" exitCode=0 Oct 30 21:55:24.421859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.421698 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" Oct 30 21:55:24.421859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.421730 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" event={"ID":"c67bb07b-9ee6-412a-a850-c872cb33b1b2","Type":"ContainerDied","Data":"80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a"} Oct 30 21:55:24.421859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.421784 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm" event={"ID":"c67bb07b-9ee6-412a-a850-c872cb33b1b2","Type":"ContainerDied","Data":"5e967fb86777611188847d7a2e7de6258bb06b6ecbd3ddc80b3da2649cb9f843"} Oct 30 21:55:24.421859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.421806 4526 scope.go:122] "RemoveContainer" containerID="80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a" Oct 30 21:55:24.433187 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.433169 4526 scope.go:122] "RemoveContainer" containerID="06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544" Oct 30 21:55:24.444098 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.444065 4526 scope.go:122] "RemoveContainer" containerID="80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a" Oct 30 21:55:24.444449 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:24.444429 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a\": container with ID starting with 80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a not found: ID does not exist" containerID="80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a" Oct 30 21:55:24.444574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.444459 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a"} err="failed to get container status \"80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a\": rpc error: code = NotFound desc = could not find container \"80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a\": container with ID starting with 80740b234c776395f3bf05284cbe02b641ee0d417ef7d8e06781b061761b139a not found: ID does not exist" Oct 30 21:55:24.444574 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.444477 4526 scope.go:122] "RemoveContainer" containerID="06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544" Oct 30 21:55:24.444793 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:24.444774 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544\": container with ID starting with 06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544 not found: ID does not exist" containerID="06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544" Oct 30 21:55:24.444849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.444800 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544"} err="failed to get container status \"06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544\": rpc error: code = NotFound desc = could not find container \"06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544\": container with ID starting with 06ab0c2276908216c2de540a42aa5c79ecaa8c9f96104ea012354b17837da544 not found: ID does not exist" Oct 30 21:55:24.487107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.487066 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.487107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.487102 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2748\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-kube-api-access-q2748\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.487107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.487117 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.487426 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.487126 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c67bb07b-9ee6-412a-a850-c872cb33b1b2-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:24.583364 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.583313 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm"] Oct 30 21:55:24.587094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:24.587062 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-013fb2-openshift-default-bc7cfdb5c-xcjdm"] Oct 30 21:55:25.870706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:25.870657 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" podUID="0e185477-5e52-45e4-9500-c04371ba67e2" containerName="payload-processing" containerID="cri-o://17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4" gracePeriod=25 Oct 30 21:55:25.986286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:25.986252 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" path="/var/lib/kubelet/pods/c67bb07b-9ee6-412a-a850-c872cb33b1b2/volumes" Oct 30 21:55:26.133704 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.133643 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:55:26.206135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.206100 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e185477-5e52-45e4-9500-c04371ba67e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e185477-5e52-45e4-9500-c04371ba67e2-plugins-config-volume\") pod \"0e185477-5e52-45e4-9500-c04371ba67e2\" (UID: \"0e185477-5e52-45e4-9500-c04371ba67e2\") " Oct 30 21:55:26.206327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.206148 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e185477-5e52-45e4-9500-c04371ba67e2-kube-api-access-jrsw5\" (UniqueName: \"kubernetes.io/projected/0e185477-5e52-45e4-9500-c04371ba67e2-kube-api-access-jrsw5\") pod \"0e185477-5e52-45e4-9500-c04371ba67e2\" (UID: \"0e185477-5e52-45e4-9500-c04371ba67e2\") " Oct 30 21:55:26.206609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.206582 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e185477-5e52-45e4-9500-c04371ba67e2-plugins-config-volume" pod "0e185477-5e52-45e4-9500-c04371ba67e2" (UID: "0e185477-5e52-45e4-9500-c04371ba67e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:26.208557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.208536 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e185477-5e52-45e4-9500-c04371ba67e2-kube-api-access-jrsw5" pod "0e185477-5e52-45e4-9500-c04371ba67e2" (UID: "0e185477-5e52-45e4-9500-c04371ba67e2"). InnerVolumeSpecName "kube-api-access-jrsw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:26.308039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.307996 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e185477-5e52-45e4-9500-c04371ba67e2-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:26.308039 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.308031 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrsw5\" (UniqueName: \"kubernetes.io/projected/0e185477-5e52-45e4-9500-c04371ba67e2-kube-api-access-jrsw5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:26.436969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.436881 4526 generic.go:356] "Generic (PLEG): container finished" podID="0e185477-5e52-45e4-9500-c04371ba67e2" containerID="17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4" exitCode=0 Oct 30 21:55:26.436969 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.436925 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" Oct 30 21:55:26.437216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.436974 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" event={"ID":"0e185477-5e52-45e4-9500-c04371ba67e2","Type":"ContainerDied","Data":"17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4"} Oct 30 21:55:26.437216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.437036 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt" event={"ID":"0e185477-5e52-45e4-9500-c04371ba67e2","Type":"ContainerDied","Data":"ff247c8b65d4c35394c1ebf67b0b161f8063eda5015e34746452bab80dc2134c"} Oct 30 21:55:26.437216 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.437067 4526 scope.go:122] "RemoveContainer" containerID="17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4" Oct 30 21:55:26.447345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.447324 4526 scope.go:122] "RemoveContainer" containerID="17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4" Oct 30 21:55:26.447700 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:26.447679 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4\": container with ID starting with 17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4 not found: ID does not exist" containerID="17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4" Oct 30 21:55:26.447762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.447710 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4"} err="failed to get container status \"17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4\": rpc error: code = NotFound desc = could not find container \"17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4\": container with ID starting with 17d136ecb61cb7ccd7fcc1d423be2a2ee93c407a0149098a4c5b76d95b4f1cf4 not found: ID does not exist" Oct 30 21:55:26.601581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.601542 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt"] Oct 30 21:55:26.608641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:26.608603 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ddec2-56c75946c9-rndgt"] Oct 30 21:55:27.985438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:27.985405 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e185477-5e52-45e4-9500-c04371ba67e2" path="/var/lib/kubelet/pods/0e185477-5e52-45e4-9500-c04371ba67e2/volumes" Oct 30 21:55:30.106988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.106944 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl"] Oct 30 21:55:30.107552 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.107535 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" Oct 30 21:55:30.107606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.107556 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" Oct 30 21:55:30.107606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.107580 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0e185477-5e52-45e4-9500-c04371ba67e2" containerName="payload-processing" Oct 30 21:55:30.107606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.107585 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e185477-5e52-45e4-9500-c04371ba67e2" containerName="payload-processing" Oct 30 21:55:30.107606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.107593 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" Oct 30 21:55:30.107606 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.107598 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" Oct 30 21:55:30.107761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.107714 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" Oct 30 21:55:30.107761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.107727 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e185477-5e52-45e4-9500-c04371ba67e2" containerName="payload-processing" Oct 30 21:55:30.112519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.112502 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:30.162192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.162159 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a38be-308b-4893-a460-451e17313fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl\" (UID: \"268a38be-308b-4893-a460-451e17313fac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:30.162192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.162197 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mtr\" (UniqueName: \"kubernetes.io/projected/268a38be-308b-4893-a460-451e17313fac-kube-api-access-t2mtr\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl\" (UID: \"268a38be-308b-4893-a460-451e17313fac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:30.173280 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.173247 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl"] Oct 30 21:55:30.221882 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.221845 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-75c2acf6-dockercfg-vjhjr\"" Oct 30 21:55:30.222192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.222175 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-75c2acf6\"" Oct 30 21:55:30.272993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.272959 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh"] Oct 30 21:55:30.273687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.273667 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67bb07b-9ee6-412a-a850-c872cb33b1b2" containerName="istio-proxy" Oct 30 21:55:30.274267 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.274240 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a38be-308b-4893-a460-451e17313fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl\" (UID: \"268a38be-308b-4893-a460-451e17313fac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:30.274367 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.274276 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mtr\" (UniqueName: \"kubernetes.io/projected/268a38be-308b-4893-a460-451e17313fac-kube-api-access-t2mtr\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl\" (UID: \"268a38be-308b-4893-a460-451e17313fac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:30.275193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.275171 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a38be-308b-4893-a460-451e17313fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl\" (UID: \"268a38be-308b-4893-a460-451e17313fac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:30.277835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.277818 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:30.323155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.323121 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh"] Oct 30 21:55:30.332436 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.332407 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mtr\" (UniqueName: \"kubernetes.io/projected/268a38be-308b-4893-a460-451e17313fac-kube-api-access-t2mtr\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl\" (UID: \"268a38be-308b-4893-a460-451e17313fac\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:30.375603 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.375482 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-plugins-config-volume\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh\" (UID: \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:30.375770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.375597 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwhpd\" (UniqueName: \"kubernetes.io/projected/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-kube-api-access-zwhpd\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh\" (UID: \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:30.427031 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.426999 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:30.476638 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.476582 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-plugins-config-volume\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh\" (UID: \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:30.476824 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.476666 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwhpd\" (UniqueName: \"kubernetes.io/projected/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-kube-api-access-zwhpd\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh\" (UID: \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:30.477327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.477307 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-plugins-config-volume\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh\" (UID: \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:30.533811 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.533775 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwhpd\" (UniqueName: \"kubernetes.io/projected/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-kube-api-access-zwhpd\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh\" (UID: \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:30.591694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.591653 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:30.808670 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.808645 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl"] Oct 30 21:55:30.810056 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:55:30.810030 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268a38be_308b_4893_a460_451e17313fac.slice/crio-3d0b4903846a2640a68821c3a2c1aed0eeccaccad0c59520610dadcdd9c05814 WatchSource:0}: Error finding container 3d0b4903846a2640a68821c3a2c1aed0eeccaccad0c59520610dadcdd9c05814: Status 404 returned error can't find the container with id 3d0b4903846a2640a68821c3a2c1aed0eeccaccad0c59520610dadcdd9c05814 Oct 30 21:55:30.982020 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:30.981995 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh"] Oct 30 21:55:30.982676 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:55:30.982653 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a617e0_ca59_4e58_8e57_b2dbaa6b928b.slice/crio-49171d3b23e7ac7791ea55ce2dd9dfbe9f21bdc50633c9684fdae3aa4711b0d9 WatchSource:0}: Error finding container 49171d3b23e7ac7791ea55ce2dd9dfbe9f21bdc50633c9684fdae3aa4711b0d9: Status 404 returned error can't find the container with id 49171d3b23e7ac7791ea55ce2dd9dfbe9f21bdc50633c9684fdae3aa4711b0d9 Oct 30 21:55:31.473217 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:31.473186 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" event={"ID":"268a38be-308b-4893-a460-451e17313fac","Type":"ContainerStarted","Data":"785c2684ee8c5c16ae6289dbd2db3bd7f7400045a0db43cd2b6abdadd5c532ba"} Oct 30 21:55:31.473747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:31.473237 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" event={"ID":"268a38be-308b-4893-a460-451e17313fac","Type":"ContainerStarted","Data":"3d0b4903846a2640a68821c3a2c1aed0eeccaccad0c59520610dadcdd9c05814"} Oct 30 21:55:31.473747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:31.473400 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:31.475050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:31.475027 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" event={"ID":"51a617e0-ca59-4e58-8e57-b2dbaa6b928b","Type":"ContainerStarted","Data":"323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5"} Oct 30 21:55:31.475173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:31.475068 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:31.475173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:31.475084 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" event={"ID":"51a617e0-ca59-4e58-8e57-b2dbaa6b928b","Type":"ContainerStarted","Data":"49171d3b23e7ac7791ea55ce2dd9dfbe9f21bdc50633c9684fdae3aa4711b0d9"} Oct 30 21:55:31.590536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:31.590461 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" podStartSLOduration=1.590446193 podStartE2EDuration="1.590446193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:31.587846682 +0000 UTC m=+2321.721219882" watchObservedRunningTime="2026-10-30 21:55:31.590446193 +0000 UTC m=+2321.723819390" Oct 30 21:55:31.708898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:31.708843 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" podStartSLOduration=1.7088267209999999 podStartE2EDuration="1.708826721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:31.703994371 +0000 UTC m=+2321.837367571" watchObservedRunningTime="2026-10-30 21:55:31.708826721 +0000 UTC m=+2321.842199920" Oct 30 21:55:42.478838 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:55:42.478800 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:55:42.484684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:42.484665 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 21:55:42.484836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:42.484726 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 21:55:51.968467 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:51.968434 4526 scope.go:122] "RemoveContainer" containerID="b0372f3b422f02ecac50e2846bf20011c68458362989fba8ae78964fe0e8f275" Oct 30 21:55:54.034390 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.034353 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv"] Oct 30 21:55:54.040016 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.039996 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.073791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.073760 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m"] Oct 30 21:55:54.077767 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.077748 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.093477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.093455 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:55:54.122802 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.122772 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv"] Oct 30 21:55:54.144156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.144127 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m"] Oct 30 21:55:54.180208 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.180170 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.180381 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.180218 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvztm\" (UniqueName: \"kubernetes.io/projected/12a2fe3e-8217-4535-9e8f-596ff75e2352-kube-api-access-zvztm\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.180459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.180385 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.180597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.180566 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlv6f\" (UniqueName: \"kubernetes.io/projected/68150331-c4b4-4c27-9a89-be2a545153c2-kube-api-access-hlv6f\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.180698 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.180648 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.180796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.180777 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.193757 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.193717 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:54.282257 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.282227 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.282465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.282346 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlv6f\" (UniqueName: \"kubernetes.io/projected/68150331-c4b4-4c27-9a89-be2a545153c2-kube-api-access-hlv6f\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.282465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.282396 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.282653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.282463 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.282653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.282548 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.282653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.282575 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvztm\" (UniqueName: \"kubernetes.io/projected/12a2fe3e-8217-4535-9e8f-596ff75e2352-kube-api-access-zvztm\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.283052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.283030 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.283142 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.283097 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.285152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.285099 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.285152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.285106 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.340636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.340601 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvztm\" (UniqueName: \"kubernetes.io/projected/12a2fe3e-8217-4535-9e8f-596ff75e2352-kube-api-access-zvztm\") pod \"maas-api-revoke-sub-2fc0d783-k6z8m\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.340745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.340717 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlv6f\" (UniqueName: \"kubernetes.io/projected/68150331-c4b4-4c27-9a89-be2a545153c2-kube-api-access-hlv6f\") pod \"maas-api-revoke-sub-afa324b1-2pdvv\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.353477 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.353452 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:54.391038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.391005 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:54.756267 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.756237 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv"] Oct 30 21:55:54.757683 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:55:54.757660 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68150331_c4b4_4c27_9a89_be2a545153c2.slice/crio-49826af3458d9c1feefb46c7dcd6b81d742b6598c45b25bdbee197692c9cb05f WatchSource:0}: Error finding container 49826af3458d9c1feefb46c7dcd6b81d742b6598c45b25bdbee197692c9cb05f: Status 404 returned error can't find the container with id 49826af3458d9c1feefb46c7dcd6b81d742b6598c45b25bdbee197692c9cb05f Oct 30 21:55:54.808382 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:54.808351 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m"] Oct 30 21:55:54.808880 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:55:54.808858 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a2fe3e_8217_4535_9e8f_596ff75e2352.slice/crio-f13f9309c5a45721da879b21c10f80a309cf2c47c2c6592f8a864221bf86a121 WatchSource:0}: Error finding container f13f9309c5a45721da879b21c10f80a309cf2c47c2c6592f8a864221bf86a121: Status 404 returned error can't find the container with id f13f9309c5a45721da879b21c10f80a309cf2c47c2c6592f8a864221bf86a121 Oct 30 21:55:55.636448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:55.636414 4526 generic.go:356] "Generic (PLEG): container finished" podID="68150331-c4b4-4c27-9a89-be2a545153c2" containerID="afa2bd5e6210e1149d76fcaf9d0a9d1699ec53a34e1dc58ce2b4880eb406f881" exitCode=0 Oct 30 21:55:55.636891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:55.636504 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" event={"ID":"68150331-c4b4-4c27-9a89-be2a545153c2","Type":"ContainerDied","Data":"afa2bd5e6210e1149d76fcaf9d0a9d1699ec53a34e1dc58ce2b4880eb406f881"} Oct 30 21:55:55.636891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:55.636567 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" event={"ID":"68150331-c4b4-4c27-9a89-be2a545153c2","Type":"ContainerStarted","Data":"49826af3458d9c1feefb46c7dcd6b81d742b6598c45b25bdbee197692c9cb05f"} Oct 30 21:55:55.638445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:55.638413 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" event={"ID":"12a2fe3e-8217-4535-9e8f-596ff75e2352","Type":"ContainerStarted","Data":"1260edd8529f3c823b82df0dd83c184718a5ebf6c6128d4ff83a2f3e21563627"} Oct 30 21:55:55.638610 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:55.638469 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" event={"ID":"12a2fe3e-8217-4535-9e8f-596ff75e2352","Type":"ContainerStarted","Data":"f13f9309c5a45721da879b21c10f80a309cf2c47c2c6592f8a864221bf86a121"} Oct 30 21:55:55.872636 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:55.872514 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" podStartSLOduration=1.872482591 podStartE2EDuration="1.872482591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:55.871003593 +0000 UTC m=+2346.004376804" watchObservedRunningTime="2026-10-30 21:55:55.872482591 +0000 UTC m=+2346.005855825" Oct 30 21:55:56.371448 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.371409 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld"] Oct 30 21:55:56.378077 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.378049 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-89c0fd-9gjld"] Oct 30 21:55:56.647423 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.647336 4526 generic.go:356] "Generic (PLEG): container finished" podID="12a2fe3e-8217-4535-9e8f-596ff75e2352" containerID="1260edd8529f3c823b82df0dd83c184718a5ebf6c6128d4ff83a2f3e21563627" exitCode=0 Oct 30 21:55:56.647907 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.647429 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" event={"ID":"12a2fe3e-8217-4535-9e8f-596ff75e2352","Type":"ContainerDied","Data":"1260edd8529f3c823b82df0dd83c184718a5ebf6c6128d4ff83a2f3e21563627"} Oct 30 21:55:56.823810 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.823779 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:56.925659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.925570 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-xks-ca\") pod \"68150331-c4b4-4c27-9a89-be2a545153c2\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " Oct 30 21:55:56.925659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.925631 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-openshift-ca\") pod \"68150331-c4b4-4c27-9a89-be2a545153c2\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " Oct 30 21:55:56.925899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.925729 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68150331-c4b4-4c27-9a89-be2a545153c2-kube-api-access-hlv6f\" (UniqueName: \"kubernetes.io/projected/68150331-c4b4-4c27-9a89-be2a545153c2-kube-api-access-hlv6f\") pod \"68150331-c4b4-4c27-9a89-be2a545153c2\" (UID: \"68150331-c4b4-4c27-9a89-be2a545153c2\") " Oct 30 21:55:56.925981 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.925957 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-openshift-ca" pod "68150331-c4b4-4c27-9a89-be2a545153c2" (UID: "68150331-c4b4-4c27-9a89-be2a545153c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:56.926344 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.926300 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:56.927852 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.927827 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-xks-ca" pod "68150331-c4b4-4c27-9a89-be2a545153c2" (UID: "68150331-c4b4-4c27-9a89-be2a545153c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:56.927953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:56.927919 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68150331-c4b4-4c27-9a89-be2a545153c2-kube-api-access-hlv6f" pod "68150331-c4b4-4c27-9a89-be2a545153c2" (UID: "68150331-c4b4-4c27-9a89-be2a545153c2"). InnerVolumeSpecName "kube-api-access-hlv6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:57.027116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.027083 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlv6f\" (UniqueName: \"kubernetes.io/projected/68150331-c4b4-4c27-9a89-be2a545153c2-kube-api-access-hlv6f\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:57.027116 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.027113 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68150331-c4b4-4c27-9a89-be2a545153c2-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:57.656242 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.656208 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" Oct 30 21:55:57.656706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.656235 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv" event={"ID":"68150331-c4b4-4c27-9a89-be2a545153c2","Type":"ContainerDied","Data":"49826af3458d9c1feefb46c7dcd6b81d742b6598c45b25bdbee197692c9cb05f"} Oct 30 21:55:57.656706 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.656291 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49826af3458d9c1feefb46c7dcd6b81d742b6598c45b25bdbee197692c9cb05f" Oct 30 21:55:57.844268 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.844235 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw"] Oct 30 21:55:57.853057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.853029 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34958-c8vgw"] Oct 30 21:55:57.862463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.862443 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:57.942418 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.942314 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-openshift-ca\") pod \"12a2fe3e-8217-4535-9e8f-596ff75e2352\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " Oct 30 21:55:57.942627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.942453 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a2fe3e-8217-4535-9e8f-596ff75e2352-kube-api-access-zvztm\" (UniqueName: \"kubernetes.io/projected/12a2fe3e-8217-4535-9e8f-596ff75e2352-kube-api-access-zvztm\") pod \"12a2fe3e-8217-4535-9e8f-596ff75e2352\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " Oct 30 21:55:57.942627 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.942580 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-xks-ca\") pod \"12a2fe3e-8217-4535-9e8f-596ff75e2352\" (UID: \"12a2fe3e-8217-4535-9e8f-596ff75e2352\") " Oct 30 21:55:57.942761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.942736 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-openshift-ca" pod "12a2fe3e-8217-4535-9e8f-596ff75e2352" (UID: "12a2fe3e-8217-4535-9e8f-596ff75e2352"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:57.942997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.942976 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:57.944808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.944786 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-xks-ca" pod "12a2fe3e-8217-4535-9e8f-596ff75e2352" (UID: "12a2fe3e-8217-4535-9e8f-596ff75e2352"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:57.944808 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.944801 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a2fe3e-8217-4535-9e8f-596ff75e2352-kube-api-access-zvztm" pod "12a2fe3e-8217-4535-9e8f-596ff75e2352" (UID: "12a2fe3e-8217-4535-9e8f-596ff75e2352"). InnerVolumeSpecName "kube-api-access-zvztm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:57.985887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.985847 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078567c1-6c75-4bf7-91f4-046a7c3450bc" path="/var/lib/kubelet/pods/078567c1-6c75-4bf7-91f4-046a7c3450bc/volumes" Oct 30 21:55:57.986397 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:57.986374 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48764c5-8809-486d-90ab-d761bd31fcd1" path="/var/lib/kubelet/pods/e48764c5-8809-486d-90ab-d761bd31fcd1/volumes" Oct 30 21:55:58.044154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:58.044118 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvztm\" (UniqueName: \"kubernetes.io/projected/12a2fe3e-8217-4535-9e8f-596ff75e2352-kube-api-access-zvztm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:58.044154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:58.044150 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a2fe3e-8217-4535-9e8f-596ff75e2352-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:58.664877 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:58.664848 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" Oct 30 21:55:58.665273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:58.664846 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m" event={"ID":"12a2fe3e-8217-4535-9e8f-596ff75e2352","Type":"ContainerDied","Data":"f13f9309c5a45721da879b21c10f80a309cf2c47c2c6592f8a864221bf86a121"} Oct 30 21:55:58.665273 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:58.664999 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13f9309c5a45721da879b21c10f80a309cf2c47c2c6592f8a864221bf86a121" Oct 30 21:55:59.084539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:59.084471 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7"] Oct 30 21:55:59.087453 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:59.087427 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ddec2-npzg7"] Oct 30 21:55:59.986044 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:55:59.986009 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a13c39-65d4-4e4e-8d8e-fa90e603edc0" path="/var/lib/kubelet/pods/38a13c39-65d4-4e4e-8d8e-fa90e603edc0/volumes" Oct 30 21:56:05.483210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:05.483168 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz"] Oct 30 21:56:05.483720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:05.483481 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" containerID="cri-o://55f5cc51c894fd7e376cde387c8a4879bf143299876fdbea38c640f5b81ceeaf" gracePeriod=30 Oct 30 21:56:10.751272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.751202 4526 generic.go:356] "Generic (PLEG): container finished" podID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerID="55f5cc51c894fd7e376cde387c8a4879bf143299876fdbea38c640f5b81ceeaf" exitCode=0 Oct 30 21:56:10.751626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.751284 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" event={"ID":"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f","Type":"ContainerDied","Data":"55f5cc51c894fd7e376cde387c8a4879bf143299876fdbea38c640f5b81ceeaf"} Oct 30 21:56:10.751626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.751350 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" event={"ID":"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f","Type":"ContainerDied","Data":"39ee059d681d12163b0334a8f0f804138aae6190954adc41b228c194b4dd4afd"} Oct 30 21:56:10.751626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.751374 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ee059d681d12163b0334a8f0f804138aae6190954adc41b228c194b4dd4afd" Oct 30 21:56:10.764733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.764710 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:56:10.903558 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.903525 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp"] Oct 30 21:56:10.903755 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.903707 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-certs\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.903842 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.903828 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-kube-api-access-rk2k9\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-kube-api-access-rk2k9\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.903894 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.903875 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-token\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-token\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.903957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.903913 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-credential-socket\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.904024 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.903964 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-socket\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.904024 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.903983 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-certs" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.904024 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.903994 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-podinfo\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.904199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904095 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istiod-ca-cert\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.904199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904138 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-envoy\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.904199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904168 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-data\") pod \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\" (UID: \"ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f\") " Oct 30 21:56:10.904476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904454 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-credential-socket" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.904577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904467 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-socket" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.904577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904543 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istiod-ca-cert" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:10.904748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904725 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-data" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.904847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904827 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.905012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904852 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.905012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904863 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.905012 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.904872 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.906662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.906628 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-token" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.906818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.906688 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-envoy" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.906818 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.906777 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-kube-api-access-rk2k9" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "kube-api-access-rk2k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.907097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:10.907077 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-podinfo" pod "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" (UID: "ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:11.006590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.006516 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk2k9\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-kube-api-access-rk2k9\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:11.006590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.006544 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:11.006590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.006553 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:11.006590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.006562 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:11.006590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.006571 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:11.702509 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.702437 4526 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:56:11.702673 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.702535 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:56:11.758259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.758228 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz" Oct 30 21:56:11.926090 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.926055 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz"] Oct 30 21:56:11.932520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.932497 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-blkpz"] Oct 30 21:56:11.993588 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:11.993512 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" path="/var/lib/kubelet/pods/ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f/volumes" Oct 30 21:56:12.478861 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:56:12.478827 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:56:16.497875 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.497847 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz"] Oct 30 21:56:16.498388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498369 4526 cpu_manager.go:414] "containerMap: removing container" podUID="68150331-c4b4-4c27-9a89-be2a545153c2" containerName="revoke-keys" Oct 30 21:56:16.498388 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498384 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68150331-c4b4-4c27-9a89-be2a545153c2" containerName="revoke-keys" Oct 30 21:56:16.498466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498403 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" Oct 30 21:56:16.498466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498408 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" Oct 30 21:56:16.498466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498419 4526 cpu_manager.go:414] "containerMap: removing container" podUID="12a2fe3e-8217-4535-9e8f-596ff75e2352" containerName="revoke-keys" Oct 30 21:56:16.498466 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498424 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a2fe3e-8217-4535-9e8f-596ff75e2352" containerName="revoke-keys" Oct 30 21:56:16.498615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498516 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="68150331-c4b4-4c27-9a89-be2a545153c2" containerName="revoke-keys" Oct 30 21:56:16.498615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498528 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb6d36a-acfa-410e-b3e3-c8fc1102ec1f" containerName="istio-proxy" Oct 30 21:56:16.498615 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.498535 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a2fe3e-8217-4535-9e8f-596ff75e2352" containerName="revoke-keys" Oct 30 21:56:16.501781 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.501767 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.554391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.554360 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:16.558616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.558585 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz"] Oct 30 21:56:16.603617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.603587 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:56:16.671731 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.671688 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.671892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.671761 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.671892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.671860 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrnn\" (UniqueName: \"kubernetes.io/projected/b803d94d-e17b-4514-bdd8-8b2066f01eca-kube-api-access-jsrnn\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.773691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.773599 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.773691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.773652 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.773691 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.773683 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrnn\" (UniqueName: \"kubernetes.io/projected/b803d94d-e17b-4514-bdd8-8b2066f01eca-kube-api-access-jsrnn\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.774327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.774303 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.776233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.776211 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:16.830222 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:16.830194 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrnn\" (UniqueName: \"kubernetes.io/projected/b803d94d-e17b-4514-bdd8-8b2066f01eca-kube-api-access-jsrnn\") pod \"maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:17.115111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:17.115080 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:17.498772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:17.498743 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz"] Oct 30 21:56:17.501209 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:56:17.501185 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb803d94d_e17b_4514_bdd8_8b2066f01eca.slice/crio-184c935664ec7bb057afc717f0f595312777b521b39f9241bd3da53297617695 WatchSource:0}: Error finding container 184c935664ec7bb057afc717f0f595312777b521b39f9241bd3da53297617695: Status 404 returned error can't find the container with id 184c935664ec7bb057afc717f0f595312777b521b39f9241bd3da53297617695 Oct 30 21:56:17.801770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:17.801733 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" event={"ID":"b803d94d-e17b-4514-bdd8-8b2066f01eca","Type":"ContainerStarted","Data":"184c935664ec7bb057afc717f0f595312777b521b39f9241bd3da53297617695"} Oct 30 21:56:18.809742 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:18.809707 4526 generic.go:356] "Generic (PLEG): container finished" podID="b803d94d-e17b-4514-bdd8-8b2066f01eca" containerID="b0d196762493e495c6548925ce2c827e571497199e6bbab86fe28d30ea3ed0bd" exitCode=0 Oct 30 21:56:18.810183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:18.809802 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" event={"ID":"b803d94d-e17b-4514-bdd8-8b2066f01eca","Type":"ContainerDied","Data":"b0d196762493e495c6548925ce2c827e571497199e6bbab86fe28d30ea3ed0bd"} Oct 30 21:56:19.141133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.141019 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh"] Oct 30 21:56:19.145205 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.145183 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.197307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.197277 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dbe6da-openshift-default-dockercfg-jcr76\"" Oct 30 21:56:19.208063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.208031 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh"] Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308227 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-credential-socket\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308292 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-data\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308330 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-certs\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308419 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-podinfo\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308458 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-socket\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308544 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-token\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308604 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzw2z\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-kube-api-access-kzw2z\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308656 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istiod-ca-cert\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.326789 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.308687 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-envoy\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.421600 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.421447 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-credential-socket\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.422651 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.422626 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-data\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.422855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.422200 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-credential-socket\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.422980 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.422847 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-certs\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.423071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.423052 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-data\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.423171 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.423151 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-podinfo\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.423248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.423200 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-socket\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.423302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.423248 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-certs\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.423302 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.423261 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-token\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.423409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.423327 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzw2z\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-kube-api-access-kzw2z\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.423409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.423370 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istiod-ca-cert\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.424094 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.424066 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-socket\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.424190 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.424128 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-envoy\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.424901 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.424877 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istiod-ca-cert\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.427378 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.427335 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-envoy\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.434660 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.434624 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-podinfo\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.481010 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.480973 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-token\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.481199 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.481088 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzw2z\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-kube-api-access-kzw2z\") pod \"e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:19.759753 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:19.759587 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:20.039182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.039156 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:20.134047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.134005 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-xks-ca\") pod \"b803d94d-e17b-4514-bdd8-8b2066f01eca\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " Oct 30 21:56:20.134252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.134065 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b803d94d-e17b-4514-bdd8-8b2066f01eca-kube-api-access-jsrnn\" (UniqueName: \"kubernetes.io/projected/b803d94d-e17b-4514-bdd8-8b2066f01eca-kube-api-access-jsrnn\") pod \"b803d94d-e17b-4514-bdd8-8b2066f01eca\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " Oct 30 21:56:20.134252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.134120 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-openshift-ca\") pod \"b803d94d-e17b-4514-bdd8-8b2066f01eca\" (UID: \"b803d94d-e17b-4514-bdd8-8b2066f01eca\") " Oct 30 21:56:20.134542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.134479 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-openshift-ca" pod "b803d94d-e17b-4514-bdd8-8b2066f01eca" (UID: "b803d94d-e17b-4514-bdd8-8b2066f01eca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:20.136359 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.136334 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b803d94d-e17b-4514-bdd8-8b2066f01eca-kube-api-access-jsrnn" pod "b803d94d-e17b-4514-bdd8-8b2066f01eca" (UID: "b803d94d-e17b-4514-bdd8-8b2066f01eca"). InnerVolumeSpecName "kube-api-access-jsrnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:20.136463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.136377 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-xks-ca" pod "b803d94d-e17b-4514-bdd8-8b2066f01eca" (UID: "b803d94d-e17b-4514-bdd8-8b2066f01eca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:20.205313 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.205276 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh"] Oct 30 21:56:20.213997 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:56:20.213969 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351cddf1_72d9_404d_b1e8_d1592a4bffe8.slice/crio-67b5089cb6202a531a08f08636e29a32c9b7de42ab9bc70f7a177116ad778880 WatchSource:0}: Error finding container 67b5089cb6202a531a08f08636e29a32c9b7de42ab9bc70f7a177116ad778880: Status 404 returned error can't find the container with id 67b5089cb6202a531a08f08636e29a32c9b7de42ab9bc70f7a177116ad778880 Oct 30 21:56:20.216765 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.216732 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:20.216867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.216799 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:20.216867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.216829 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:20.235960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.235934 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:20.235960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.235962 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrnn\" (UniqueName: \"kubernetes.io/projected/b803d94d-e17b-4514-bdd8-8b2066f01eca-kube-api-access-jsrnn\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:20.236132 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.235971 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b803d94d-e17b-4514-bdd8-8b2066f01eca-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:20.827312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.827276 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" event={"ID":"351cddf1-72d9-404d-b1e8-d1592a4bffe8","Type":"ContainerStarted","Data":"fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d"} Oct 30 21:56:20.827584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.827364 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" event={"ID":"351cddf1-72d9-404d-b1e8-d1592a4bffe8","Type":"ContainerStarted","Data":"67b5089cb6202a531a08f08636e29a32c9b7de42ab9bc70f7a177116ad778880"} Oct 30 21:56:20.828959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.828938 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" Oct 30 21:56:20.829079 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.828976 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz" event={"ID":"b803d94d-e17b-4514-bdd8-8b2066f01eca","Type":"ContainerDied","Data":"184c935664ec7bb057afc717f0f595312777b521b39f9241bd3da53297617695"} Oct 30 21:56:20.829079 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.829019 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="184c935664ec7bb057afc717f0f595312777b521b39f9241bd3da53297617695" Oct 30 21:56:20.944248 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:20.944191 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podStartSLOduration=1.944176889 podStartE2EDuration="1.944176889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:20.94240944 +0000 UTC m=+2371.075782648" watchObservedRunningTime="2026-10-30 21:56:20.944176889 +0000 UTC m=+2371.077550087" Oct 30 21:56:21.199869 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:21.199768 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh"] Oct 30 21:56:21.202961 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:21.202934 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1dc70e-kd8mh"] Oct 30 21:56:21.760125 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:21.760091 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:21.766255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:21.766228 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:21.835957 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:21.835928 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:21.836826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:21.836804 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:56:21.986702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:21.986669 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db567ebe-3deb-409b-b7ed-737fbd5a1518" path="/var/lib/kubelet/pods/db567ebe-3deb-409b-b7ed-737fbd5a1518/volumes" Oct 30 21:56:24.483906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:24.483862 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p"] Oct 30 21:56:24.484634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:24.484097 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" containerID="cri-o://e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507" gracePeriod=30 Oct 30 21:56:25.263643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:25.263603 4526 patch_prober.go:29] interesting pod/e2e-api-a-b34958-openshift-default-559d74f555-htq4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:25.263845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:25.263676 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:25.925771 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:25.925724 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" containerID="cri-o://a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d" gracePeriod=45 Oct 30 21:56:26.196001 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.195973 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:56:26.321856 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.321824 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-home\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-home\") pod \"8d354f8a-4ff3-466c-875f-d7cda66177e7\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " Oct 30 21:56:26.322131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.321907 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-dshm\") pod \"8d354f8a-4ff3-466c-875f-d7cda66177e7\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " Oct 30 21:56:26.322131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322013 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-tmp-dir\") pod \"8d354f8a-4ff3-466c-875f-d7cda66177e7\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " Oct 30 21:56:26.322131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322046 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d354f8a-4ff3-466c-875f-d7cda66177e7-kube-api-access-f9ldh\" (UniqueName: \"kubernetes.io/projected/8d354f8a-4ff3-466c-875f-d7cda66177e7-kube-api-access-f9ldh\") pod \"8d354f8a-4ff3-466c-875f-d7cda66177e7\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " Oct 30 21:56:26.322131 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322107 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-model-cache\") pod \"8d354f8a-4ff3-466c-875f-d7cda66177e7\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " Oct 30 21:56:26.322360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322130 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-home" pod "8d354f8a-4ff3-466c-875f-d7cda66177e7" (UID: "8d354f8a-4ff3-466c-875f-d7cda66177e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:26.322360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322238 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d354f8a-4ff3-466c-875f-d7cda66177e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d354f8a-4ff3-466c-875f-d7cda66177e7-tls-certs\") pod \"8d354f8a-4ff3-466c-875f-d7cda66177e7\" (UID: \"8d354f8a-4ff3-466c-875f-d7cda66177e7\") " Oct 30 21:56:26.322360 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322325 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-tmp-dir" pod "8d354f8a-4ff3-466c-875f-d7cda66177e7" (UID: "8d354f8a-4ff3-466c-875f-d7cda66177e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:26.322570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322548 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-model-cache" pod "8d354f8a-4ff3-466c-875f-d7cda66177e7" (UID: "8d354f8a-4ff3-466c-875f-d7cda66177e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:26.322840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322743 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.322840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322772 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.322840 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.322786 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.329031 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.328998 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d354f8a-4ff3-466c-875f-d7cda66177e7-kube-api-access-f9ldh" pod "8d354f8a-4ff3-466c-875f-d7cda66177e7" (UID: "8d354f8a-4ff3-466c-875f-d7cda66177e7"). InnerVolumeSpecName "kube-api-access-f9ldh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:26.330564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.330468 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d354f8a-4ff3-466c-875f-d7cda66177e7-tls-certs" pod "8d354f8a-4ff3-466c-875f-d7cda66177e7" (UID: "8d354f8a-4ff3-466c-875f-d7cda66177e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:26.335823 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.335800 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-dshm" pod "8d354f8a-4ff3-466c-875f-d7cda66177e7" (UID: "8d354f8a-4ff3-466c-875f-d7cda66177e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:26.424053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.424021 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d354f8a-4ff3-466c-875f-d7cda66177e7-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.424053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.424050 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d354f8a-4ff3-466c-875f-d7cda66177e7-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.424053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.424060 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ldh\" (UniqueName: \"kubernetes.io/projected/8d354f8a-4ff3-466c-875f-d7cda66177e7-kube-api-access-f9ldh\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.869681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.869649 4526 generic.go:356] "Generic (PLEG): container finished" podID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerID="a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d" exitCode=0 Oct 30 21:56:26.869883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.869744 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" Oct 30 21:56:26.869883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.869764 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" event={"ID":"8d354f8a-4ff3-466c-875f-d7cda66177e7","Type":"ContainerDied","Data":"a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d"} Oct 30 21:56:26.869883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.869811 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp" event={"ID":"8d354f8a-4ff3-466c-875f-d7cda66177e7","Type":"ContainerDied","Data":"08258368eb07f8e2b1206ce87b5770a999dfbc7f92deb63632fc63a40c29ea97"} Oct 30 21:56:26.869883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.869834 4526 scope.go:122] "RemoveContainer" containerID="a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d" Oct 30 21:56:26.881385 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.881334 4526 scope.go:122] "RemoveContainer" containerID="a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d" Oct 30 21:56:26.881750 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:56:26.881707 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d\": container with ID starting with a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d not found: ID does not exist" containerID="a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d" Oct 30 21:56:26.881835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:26.881765 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d"} err="failed to get container status \"a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d\": rpc error: code = NotFound desc = could not find container \"a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d\": container with ID starting with a0f14548670208c92e63a0abee49f544892a2e33c45be30b6b2126c07c68589d not found: ID does not exist" Oct 30 21:56:27.032807 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:27.032772 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp"] Oct 30 21:56:27.081982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:27.081944 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/premium-w1-e4dadd-kserve-66cc9dddb7-8thdp"] Oct 30 21:56:27.789756 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:27.789719 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs"] Oct 30 21:56:27.902016 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:27.901981 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d"] Oct 30 21:56:27.985506 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:27.985466 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" path="/var/lib/kubelet/pods/8d354f8a-4ff3-466c-875f-d7cda66177e7/volumes" Oct 30 21:56:28.227742 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:28.227633 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podUID="a4207178a950303c256d4f333088d96f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:56:29.747639 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.747616 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:56:29.893149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893116 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0121879d-c1e6-4baa-91cb-c6c492fd2317-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121879d-c1e6-4baa-91cb-c6c492fd2317-istiod-ca-cert\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893170 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-socket\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893260 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-kube-api-access-zthhg\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-kube-api-access-zthhg\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893323 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-credential-socket\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893429 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-data\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893475 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-token\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-token\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893475 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-socket" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:29.893599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893524 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-certs\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893565 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-envoy\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893608 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-podinfo\") pod \"0121879d-c1e6-4baa-91cb-c6c492fd2317\" (UID: \"0121879d-c1e6-4baa-91cb-c6c492fd2317\") " Oct 30 21:56:29.893847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893686 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-credential-socket" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:29.893847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893699 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0121879d-c1e6-4baa-91cb-c6c492fd2317-istiod-ca-cert" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:29.893847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893727 4526 generic.go:356] "Generic (PLEG): container finished" podID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerID="e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507" exitCode=0 Oct 30 21:56:29.893847 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893786 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" Oct 30 21:56:29.894119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893901 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" event={"ID":"0121879d-c1e6-4baa-91cb-c6c492fd2317","Type":"ContainerDied","Data":"e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507"} Oct 30 21:56:29.894119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893909 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-certs" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:29.894119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893944 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p" event={"ID":"0121879d-c1e6-4baa-91cb-c6c492fd2317","Type":"ContainerDied","Data":"dee9f295e5b801f1db5d12bde2824c481a6849a578e55cc3a2406400fef920b4"} Oct 30 21:56:29.894119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.893971 4526 scope.go:122] "RemoveContainer" containerID="e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507" Oct 30 21:56:29.894119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.894015 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-data" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:29.894355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.894186 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.894355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.894207 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.894355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.894221 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.894355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.894236 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121879d-c1e6-4baa-91cb-c6c492fd2317-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.894355 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.894250 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.896418 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.896394 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-token" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:29.896418 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.896407 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-podinfo" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:29.896601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.896459 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-kube-api-access-zthhg" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "kube-api-access-zthhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:29.896601 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.896583 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-envoy" pod "0121879d-c1e6-4baa-91cb-c6c492fd2317" (UID: "0121879d-c1e6-4baa-91cb-c6c492fd2317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:29.925120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.925088 4526 scope.go:122] "RemoveContainer" containerID="0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61" Oct 30 21:56:29.939526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.939326 4526 scope.go:122] "RemoveContainer" containerID="e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507" Oct 30 21:56:29.942633 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:56:29.942601 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507\": container with ID starting with e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507 not found: ID does not exist" containerID="e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507" Oct 30 21:56:29.942773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.942643 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507"} err="failed to get container status \"e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507\": rpc error: code = NotFound desc = could not find container \"e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507\": container with ID starting with e7694d577889f0b75af5825b4f2c6ca161726af2e088d7818a2bbd6a0b4e2507 not found: ID does not exist" Oct 30 21:56:29.942773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.942670 4526 scope.go:122] "RemoveContainer" containerID="0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61" Oct 30 21:56:29.943049 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:56:29.943025 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61\": container with ID starting with 0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61 not found: ID does not exist" containerID="0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61" Oct 30 21:56:29.943113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.943062 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61"} err="failed to get container status \"0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61\": rpc error: code = NotFound desc = could not find container \"0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61\": container with ID starting with 0d5ceb3e3d9c6ea07a276fd2eea55006f15f614916f358fc5012146a852eee61 not found: ID does not exist" Oct 30 21:56:29.994962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.994917 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.994962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.994945 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.994962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.994954 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121879d-c1e6-4baa-91cb-c6c492fd2317-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:29.994962 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:29.994964 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zthhg\" (UniqueName: \"kubernetes.io/projected/0121879d-c1e6-4baa-91cb-c6c492fd2317-kube-api-access-zthhg\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:30.308576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:30.308540 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p"] Oct 30 21:56:30.312449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:30.312423 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b34958-openshift-default-559d74f555-htq4p"] Oct 30 21:56:31.985577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:31.985469 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" path="/var/lib/kubelet/pods/0121879d-c1e6-4baa-91cb-c6c492fd2317/volumes" Oct 30 21:56:32.790659 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:32.790609 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" podUID="fb576fcd-2113-4ce7-92b7-852040b4aa1e" containerName="payload-processing" containerID="cri-o://bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9" gracePeriod=25 Oct 30 21:56:32.902991 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:32.902946 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" podUID="e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" containerName="payload-pre-processing" containerID="cri-o://b31041897174bef0f9f069e55cf9f9f388c6f26e5ed1b930018941d348fd012d" gracePeriod=25 Oct 30 21:56:35.274656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.274623 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg"] Oct 30 21:56:35.275165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275152 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" Oct 30 21:56:35.275210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275167 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" Oct 30 21:56:35.275210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275182 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" Oct 30 21:56:35.275210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275187 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" Oct 30 21:56:35.275210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275201 4526 cpu_manager.go:414] "containerMap: removing container" podUID="b803d94d-e17b-4514-bdd8-8b2066f01eca" containerName="revoke-keys" Oct 30 21:56:35.275210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275205 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b803d94d-e17b-4514-bdd8-8b2066f01eca" containerName="revoke-keys" Oct 30 21:56:35.275369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275215 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" Oct 30 21:56:35.275369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275220 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" Oct 30 21:56:35.275369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275305 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" Oct 30 21:56:35.275369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275314 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0121879d-c1e6-4baa-91cb-c6c492fd2317" containerName="istio-proxy" Oct 30 21:56:35.275369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275322 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="b803d94d-e17b-4514-bdd8-8b2066f01eca" containerName="revoke-keys" Oct 30 21:56:35.275369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.275328 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d354f8a-4ff3-466c-875f-d7cda66177e7" containerName="main" Oct 30 21:56:35.280285 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.280268 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.336420 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.336386 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg"] Oct 30 21:56:35.381119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.381091 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:35.381296 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.381145 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:56:35.466695 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.466650 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.466919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.466877 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.467013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.466993 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb48c\" (UniqueName: \"kubernetes.io/projected/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-kube-api-access-zb48c\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.568645 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.568602 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.568830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.568681 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb48c\" (UniqueName: \"kubernetes.io/projected/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-kube-api-access-zb48c\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.568830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.568720 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.569445 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.569423 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.571265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.571235 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.625441 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.625402 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb48c\" (UniqueName: \"kubernetes.io/projected/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-kube-api-access-zb48c\") pod \"maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:35.894207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:35.894111 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:36.273892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:36.273856 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg"] Oct 30 21:56:36.275330 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:56:36.275306 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdd914b_1e72_4d32_9a34_40e3ecb1b63d.slice/crio-14b1d92acdb8b7a63826732545c677748fe335ad50d5ae54ed87353e17f6aebd WatchSource:0}: Error finding container 14b1d92acdb8b7a63826732545c677748fe335ad50d5ae54ed87353e17f6aebd: Status 404 returned error can't find the container with id 14b1d92acdb8b7a63826732545c677748fe335ad50d5ae54ed87353e17f6aebd Oct 30 21:56:36.943314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:36.943209 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" event={"ID":"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d","Type":"ContainerStarted","Data":"f5a488409b4b0137d9c18cda3536050c69e68c15f09a0deed401e77994a57f45"} Oct 30 21:56:36.943314 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:36.943266 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" event={"ID":"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d","Type":"ContainerStarted","Data":"14b1d92acdb8b7a63826732545c677748fe335ad50d5ae54ed87353e17f6aebd"} Oct 30 21:56:37.012263 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:37.012223 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" podUID="fb576fcd-2113-4ce7-92b7-852040b4aa1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:56:37.012482 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:37.012397 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" podUID="e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:56:37.056038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:37.055984 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" podStartSLOduration=2.055970615 podStartE2EDuration="2.055970615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:37.05495136 +0000 UTC m=+2387.188324573" watchObservedRunningTime="2026-10-30 21:56:37.055970615 +0000 UTC m=+2387.189343813" Oct 30 21:56:37.700831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:37.700745 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48"] Oct 30 21:56:37.701278 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:37.700976 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" containerID="cri-o://5b26801eec8a7de1d7e405077e8e301414efde171672edbd2b195257f3d6383f" gracePeriod=30 Oct 30 21:56:37.952775 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:37.952679 4526 generic.go:356] "Generic (PLEG): container finished" podID="9cdd914b-1e72-4d32-9a34-40e3ecb1b63d" containerID="f5a488409b4b0137d9c18cda3536050c69e68c15f09a0deed401e77994a57f45" exitCode=0 Oct 30 21:56:37.952959 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:37.952772 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" event={"ID":"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d","Type":"ContainerDied","Data":"f5a488409b4b0137d9c18cda3536050c69e68c15f09a0deed401e77994a57f45"} Oct 30 21:56:38.141556 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.141522 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8"] Oct 30 21:56:38.145573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.145553 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:38.196277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.196229 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54s2\" (UniqueName: \"kubernetes.io/projected/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-kube-api-access-x54s2\") pod \"payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8\" (UID: \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:38.196517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.196306 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8\" (UID: \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:38.200259 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.200231 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dbe6da\"" Oct 30 21:56:38.204898 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.204787 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8"] Oct 30 21:56:38.283860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.283828 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dbe6da-dockercfg-sd4zf\"" Oct 30 21:56:38.297830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.297791 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54s2\" (UniqueName: \"kubernetes.io/projected/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-kube-api-access-x54s2\") pod \"payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8\" (UID: \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:38.298047 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.297872 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8\" (UID: \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:38.298768 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.298741 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8\" (UID: \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:38.356584 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.356550 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54s2\" (UniqueName: \"kubernetes.io/projected/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-kube-api-access-x54s2\") pod \"payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8\" (UID: \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:38.459762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.459672 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:38.835780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.835753 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8"] Oct 30 21:56:38.836785 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:56:38.836764 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2b81c7_e253_43bd_bdab_119d0f2abc1c.slice/crio-e2008a6c6193703072216d0994636c8a19ef3f301bab759a9bb427a224df3ffd WatchSource:0}: Error finding container e2008a6c6193703072216d0994636c8a19ef3f301bab759a9bb427a224df3ffd: Status 404 returned error can't find the container with id e2008a6c6193703072216d0994636c8a19ef3f301bab759a9bb427a224df3ffd Oct 30 21:56:38.873329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.873296 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58"] Oct 30 21:56:38.879900 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.879869 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:38.907075 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.906887 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzkd5\" (UniqueName: \"kubernetes.io/projected/8f8b1499-f78b-417c-b8cf-e85918b8f227-kube-api-access-bzkd5\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:38.907174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.907116 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:38.907238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.907204 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:38.935220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.935192 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58"] Oct 30 21:56:38.962155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.962117 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" event={"ID":"dd2b81c7-e253-43bd-bdab-119d0f2abc1c","Type":"ContainerStarted","Data":"08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff"} Oct 30 21:56:38.962380 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:38.962184 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" event={"ID":"dd2b81c7-e253-43bd-bdab-119d0f2abc1c","Type":"ContainerStarted","Data":"e2008a6c6193703072216d0994636c8a19ef3f301bab759a9bb427a224df3ffd"} Oct 30 21:56:39.008597 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.008475 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:39.008802 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.008668 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:39.008802 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.008783 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzkd5\" (UniqueName: \"kubernetes.io/projected/8f8b1499-f78b-417c-b8cf-e85918b8f227-kube-api-access-bzkd5\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:39.010642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.010618 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:39.012998 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.012967 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:39.066611 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.066579 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzkd5\" (UniqueName: \"kubernetes.io/projected/8f8b1499-f78b-417c-b8cf-e85918b8f227-kube-api-access-bzkd5\") pod \"maas-api-revoke-keys-e2e-worker-w5-021907-wzs58\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:39.103359 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.103330 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:39.106381 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.106335 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" podStartSLOduration=1.106322393 podStartE2EDuration="1.106322393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:39.101979797 +0000 UTC m=+2389.235352994" watchObservedRunningTime="2026-10-30 21:56:39.106322393 +0000 UTC m=+2389.239695619" Oct 30 21:56:39.211928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.211890 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-openshift-ca\") pod \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " Oct 30 21:56:39.212152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.211949 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-xks-ca\") pod \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " Oct 30 21:56:39.212152 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.212048 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-kube-api-access-zb48c\" (UniqueName: \"kubernetes.io/projected/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-kube-api-access-zb48c\") pod \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\" (UID: \"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d\") " Oct 30 21:56:39.212813 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.212480 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-openshift-ca" pod "9cdd914b-1e72-4d32-9a34-40e3ecb1b63d" (UID: "9cdd914b-1e72-4d32-9a34-40e3ecb1b63d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:39.213245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.213216 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:39.219279 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.219203 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:39.231327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.231052 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-xks-ca" pod "9cdd914b-1e72-4d32-9a34-40e3ecb1b63d" (UID: "9cdd914b-1e72-4d32-9a34-40e3ecb1b63d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:39.231327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.231155 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-kube-api-access-zb48c" pod "9cdd914b-1e72-4d32-9a34-40e3ecb1b63d" (UID: "9cdd914b-1e72-4d32-9a34-40e3ecb1b63d"). InnerVolumeSpecName "kube-api-access-zb48c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:39.314119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.314087 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:39.314119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.314119 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb48c\" (UniqueName: \"kubernetes.io/projected/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d-kube-api-access-zb48c\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:39.608669 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.608644 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58"] Oct 30 21:56:39.610251 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:56:39.610218 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8b1499_f78b_417c_b8cf_e85918b8f227.slice/crio-644454b62d25f164258236b23386d3a62b0bca8eb0c5e1abe5469f349fab5b0f WatchSource:0}: Error finding container 644454b62d25f164258236b23386d3a62b0bca8eb0c5e1abe5469f349fab5b0f: Status 404 returned error can't find the container with id 644454b62d25f164258236b23386d3a62b0bca8eb0c5e1abe5469f349fab5b0f Oct 30 21:56:39.969990 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.969955 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" event={"ID":"8f8b1499-f78b-417c-b8cf-e85918b8f227","Type":"ContainerStarted","Data":"5995eb22d8e5fef291b28d4f965cffa88a994f122d1e29559cd5488d077274b3"} Oct 30 21:56:39.970411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.970016 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" event={"ID":"8f8b1499-f78b-417c-b8cf-e85918b8f227","Type":"ContainerStarted","Data":"644454b62d25f164258236b23386d3a62b0bca8eb0c5e1abe5469f349fab5b0f"} Oct 30 21:56:39.971369 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.971348 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" Oct 30 21:56:39.971519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.971387 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg" event={"ID":"9cdd914b-1e72-4d32-9a34-40e3ecb1b63d","Type":"ContainerDied","Data":"14b1d92acdb8b7a63826732545c677748fe335ad50d5ae54ed87353e17f6aebd"} Oct 30 21:56:39.971519 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.971426 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b1d92acdb8b7a63826732545c677748fe335ad50d5ae54ed87353e17f6aebd" Oct 30 21:56:39.971699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:39.971685 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:40.084465 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:40.084400 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" podStartSLOduration=2.084386784 podStartE2EDuration="2.084386784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:40.08243306 +0000 UTC m=+2390.215806257" watchObservedRunningTime="2026-10-30 21:56:40.084386784 +0000 UTC m=+2390.217759981" Oct 30 21:56:40.979462 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:40.979423 4526 generic.go:356] "Generic (PLEG): container finished" podID="8f8b1499-f78b-417c-b8cf-e85918b8f227" containerID="5995eb22d8e5fef291b28d4f965cffa88a994f122d1e29559cd5488d077274b3" exitCode=0 Oct 30 21:56:40.979907 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:40.979523 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" event={"ID":"8f8b1499-f78b-417c-b8cf-e85918b8f227","Type":"ContainerDied","Data":"5995eb22d8e5fef291b28d4f965cffa88a994f122d1e29559cd5488d077274b3"} Oct 30 21:56:42.186849 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.186822 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:42.221108 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.221071 4526 patch_prober.go:29] interesting pod/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:42.221290 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.221136 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:42.250529 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.250416 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8b1499-f78b-417c-b8cf-e85918b8f227-kube-api-access-bzkd5\" (UniqueName: \"kubernetes.io/projected/8f8b1499-f78b-417c-b8cf-e85918b8f227-kube-api-access-bzkd5\") pod \"8f8b1499-f78b-417c-b8cf-e85918b8f227\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " Oct 30 21:56:42.250682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.250605 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-xks-ca\") pod \"8f8b1499-f78b-417c-b8cf-e85918b8f227\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " Oct 30 21:56:42.250682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.250676 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-openshift-ca\") pod \"8f8b1499-f78b-417c-b8cf-e85918b8f227\" (UID: \"8f8b1499-f78b-417c-b8cf-e85918b8f227\") " Oct 30 21:56:42.251034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.251008 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-openshift-ca" pod "8f8b1499-f78b-417c-b8cf-e85918b8f227" (UID: "8f8b1499-f78b-417c-b8cf-e85918b8f227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:42.252939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.252911 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-xks-ca" pod "8f8b1499-f78b-417c-b8cf-e85918b8f227" (UID: "8f8b1499-f78b-417c-b8cf-e85918b8f227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:42.252939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.252932 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8b1499-f78b-417c-b8cf-e85918b8f227-kube-api-access-bzkd5" pod "8f8b1499-f78b-417c-b8cf-e85918b8f227" (UID: "8f8b1499-f78b-417c-b8cf-e85918b8f227"). InnerVolumeSpecName "kube-api-access-bzkd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:42.352018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.351988 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzkd5\" (UniqueName: \"kubernetes.io/projected/8f8b1499-f78b-417c-b8cf-e85918b8f227-kube-api-access-bzkd5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:42.352018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.352013 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:42.352018 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:42.352025 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8b1499-f78b-417c-b8cf-e85918b8f227-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:42.479474 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:56:42.479441 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:56:43.021005 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.020976 4526 generic.go:356] "Generic (PLEG): container finished" podID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerID="5b26801eec8a7de1d7e405077e8e301414efde171672edbd2b195257f3d6383f" exitCode=0 Oct 30 21:56:43.021165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.021049 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" event={"ID":"fecb65c8-f340-468e-9d6a-f731af94fd50","Type":"ContainerDied","Data":"5b26801eec8a7de1d7e405077e8e301414efde171672edbd2b195257f3d6383f"} Oct 30 21:56:43.021165 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.021104 4526 scope.go:122] "RemoveContainer" containerID="52f6dad2cc18b898395aafd8c16600831b331d238c90e2bcbacee62a8d3bfdb6" Oct 30 21:56:43.022757 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.022730 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" event={"ID":"8f8b1499-f78b-417c-b8cf-e85918b8f227","Type":"ContainerDied","Data":"644454b62d25f164258236b23386d3a62b0bca8eb0c5e1abe5469f349fab5b0f"} Oct 30 21:56:43.022886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.022768 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58" Oct 30 21:56:43.022886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.022780 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644454b62d25f164258236b23386d3a62b0bca8eb0c5e1abe5469f349fab5b0f" Oct 30 21:56:43.034700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.034680 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:56:43.061120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061092 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-podinfo\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061147 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-socket\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061192 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-envoy\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061237 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-certs\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061322 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061311 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fecb65c8-f340-468e-9d6a-f731af94fd50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecb65c8-f340-468e-9d6a-f731af94fd50-istiod-ca-cert\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061395 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-data\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061439 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-kube-api-access-8n4hw\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-kube-api-access-8n4hw\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061469 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-token\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-token\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061472 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-socket" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:43.061573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061518 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-credential-socket\") pod \"fecb65c8-f340-468e-9d6a-f731af94fd50\" (UID: \"fecb65c8-f340-468e-9d6a-f731af94fd50\") " Oct 30 21:56:43.061851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061628 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-certs" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:43.061908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061846 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-credential-socket" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:43.061908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061767 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecb65c8-f340-468e-9d6a-f731af94fd50-istiod-ca-cert" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:43.062029 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.061950 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-data" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:43.062093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.062038 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.062093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.062059 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecb65c8-f340-468e-9d6a-f731af94fd50-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.062093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.062074 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.062093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.062088 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.063883 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.063859 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-podinfo" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:43.063990 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.063937 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-kube-api-access-8n4hw" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "kube-api-access-8n4hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:43.064177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.064162 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-token" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:43.064281 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.064264 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-envoy" pod "fecb65c8-f340-468e-9d6a-f731af94fd50" (UID: "fecb65c8-f340-468e-9d6a-f731af94fd50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:43.162812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.162780 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.162812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.162810 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.162812 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.162821 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4hw\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-kube-api-access-8n4hw\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.163050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.162831 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.163050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.162841 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecb65c8-f340-468e-9d6a-f731af94fd50-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:43.257065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:43.257026 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg"] Oct 30 21:56:44.030928 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:44.030899 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" Oct 30 21:56:44.031141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:44.030940 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48" event={"ID":"fecb65c8-f340-468e-9d6a-f731af94fd50","Type":"ContainerDied","Data":"e9639024f42daeb7ac7a810237be8d859ca03611123682a3434c388d8a5e6954"} Oct 30 21:56:44.031141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:44.031009 4526 scope.go:122] "RemoveContainer" containerID="5b26801eec8a7de1d7e405077e8e301414efde171672edbd2b195257f3d6383f" Oct 30 21:56:44.259913 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:44.259834 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48"] Oct 30 21:56:44.265822 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:44.265795 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1dc70e-openshift-default-769cf49787-pvf48"] Oct 30 21:56:45.985684 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:45.985649 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" path="/var/lib/kubelet/pods/fecb65c8-f340-468e-9d6a-f731af94fd50/volumes" Oct 30 21:56:47.013204 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:47.013121 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" podUID="e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:56:47.013623 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:47.013121 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" podUID="fb576fcd-2113-4ce7-92b7-852040b4aa1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:56:50.983634 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:50.983607 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 21:56:51.081236 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.081205 4526 generic.go:356] "Generic (PLEG): container finished" podID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerID="9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d" exitCode=0 Oct 30 21:56:51.081408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.081281 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" event={"ID":"90bb6a42-f6f9-47be-a5cc-ede068832de1","Type":"ContainerDied","Data":"9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d"} Oct 30 21:56:51.081793 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.081767 4526 scope.go:122] "RemoveContainer" containerID="9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d" Oct 30 21:56:51.083229 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.083205 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:51.083297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.083272 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:51.083333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.083307 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:51.426188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.426154 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2"] Oct 30 21:56:51.426955 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.426935 4526 cpu_manager.go:414] "containerMap: removing container" podUID="9cdd914b-1e72-4d32-9a34-40e3ecb1b63d" containerName="revoke-keys" Oct 30 21:56:51.426955 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.426954 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdd914b-1e72-4d32-9a34-40e3ecb1b63d" containerName="revoke-keys" Oct 30 21:56:51.427139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.426976 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" Oct 30 21:56:51.427139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.426984 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" Oct 30 21:56:51.427139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.427005 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" Oct 30 21:56:51.427139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.427013 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" Oct 30 21:56:51.427139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.427038 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8f8b1499-f78b-417c-b8cf-e85918b8f227" containerName="revoke-keys" Oct 30 21:56:51.427139 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.427045 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8b1499-f78b-417c-b8cf-e85918b8f227" containerName="revoke-keys" Oct 30 21:56:51.427424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.427180 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8b1499-f78b-417c-b8cf-e85918b8f227" containerName="revoke-keys" Oct 30 21:56:51.427424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.427202 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdd914b-1e72-4d32-9a34-40e3ecb1b63d" containerName="revoke-keys" Oct 30 21:56:51.427424 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.427216 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" Oct 30 21:56:51.431050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.431023 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.476084 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.476048 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht"] Oct 30 21:56:51.477107 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.477085 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecb65c8-f340-468e-9d6a-f731af94fd50" containerName="istio-proxy" Oct 30 21:56:51.481189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.481170 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.483103 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.483079 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:51.483298 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.483184 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:56:51.487463 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.487438 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2"] Oct 30 21:56:51.537803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.537766 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht"] Oct 30 21:56:51.589260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.589218 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.589476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.589276 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7zk\" (UniqueName: \"kubernetes.io/projected/7e377970-46c6-4a2c-ac24-eb4f14116b4a-kube-api-access-mn7zk\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.589476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.589361 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.589476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.589418 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77kb\" (UniqueName: \"kubernetes.io/projected/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-kube-api-access-f77kb\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.589476 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.589450 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.589855 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.589626 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.691232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.691141 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77kb\" (UniqueName: \"kubernetes.io/projected/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-kube-api-access-f77kb\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.691232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.691184 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.691232 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.691217 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.691581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.691292 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.691581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.691311 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7zk\" (UniqueName: \"kubernetes.io/projected/7e377970-46c6-4a2c-ac24-eb4f14116b4a-kube-api-access-mn7zk\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.691581 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.691355 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.691967 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.691942 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.692109 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.692052 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.720514 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.719965 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.722664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.721970 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.748890 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.748859 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77kb\" (UniqueName: \"kubernetes.io/projected/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-kube-api-access-f77kb\") pod \"maas-api-revoke-sub-f365ec62-rfhht\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:51.749210 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.749141 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7zk\" (UniqueName: \"kubernetes.io/projected/7e377970-46c6-4a2c-ac24-eb4f14116b4a-kube-api-access-mn7zk\") pod \"maas-api-revoke-sub-0dad53d0-x5hg2\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:51.813530 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:51.799712 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:52.029395 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:52.029302 4526 scope.go:122] "RemoveContainer" containerID="55f5cc51c894fd7e376cde387c8a4879bf143299876fdbea38c640f5b81ceeaf" Oct 30 21:56:52.048323 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:52.047392 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:52.093193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:52.093155 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" event={"ID":"90bb6a42-f6f9-47be-a5cc-ede068832de1","Type":"ContainerStarted","Data":"b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8"} Oct 30 21:56:52.225097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:52.225071 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht"] Oct 30 21:56:52.229244 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:56:52.229209 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda833c9ee_cf5b_4a50_bece_7d4e9be9d096.slice/crio-7e1b6632ce1713e754ea907fd7dca9d832ec754cdcd89719ee59e1bf1a16e515 WatchSource:0}: Error finding container 7e1b6632ce1713e754ea907fd7dca9d832ec754cdcd89719ee59e1bf1a16e515: Status 404 returned error can't find the container with id 7e1b6632ce1713e754ea907fd7dca9d832ec754cdcd89719ee59e1bf1a16e515 Oct 30 21:56:52.438526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:52.436426 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:56:52.438526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:52.437588 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:56:52.438526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:52.437649 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 21:56:52.492986 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:52.492964 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2"] Oct 30 21:56:53.129541 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.129166 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:56:53.134438 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.134409 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" event={"ID":"7e377970-46c6-4a2c-ac24-eb4f14116b4a","Type":"ContainerStarted","Data":"8cbc2430a598fff32ad44d9bb05c1c14b177e6e706d088ac0931dbfc450508af"} Oct 30 21:56:53.134569 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.134471 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" event={"ID":"7e377970-46c6-4a2c-ac24-eb4f14116b4a","Type":"ContainerStarted","Data":"cef00493c44370d3401249a8fe14a824bd2d79492e7891dc37b8e720dcbdaa4f"} Oct 30 21:56:53.136656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.136615 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" event={"ID":"a833c9ee-cf5b-4a50-bece-7d4e9be9d096","Type":"ContainerStarted","Data":"13eef02290b1ac82a1d817d773964c1c34b490b326afee98d806a25841120b89"} Oct 30 21:56:53.136791 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.136681 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" event={"ID":"a833c9ee-cf5b-4a50-bece-7d4e9be9d096","Type":"ContainerStarted","Data":"7e1b6632ce1713e754ea907fd7dca9d832ec754cdcd89719ee59e1bf1a16e515"} Oct 30 21:56:53.141260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.141237 4526 generic.go:356] "Generic (PLEG): container finished" podID="e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" containerID="b31041897174bef0f9f069e55cf9f9f388c6f26e5ed1b930018941d348fd012d" exitCode=0 Oct 30 21:56:53.141386 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.141312 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" event={"ID":"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2","Type":"ContainerDied","Data":"b31041897174bef0f9f069e55cf9f9f388c6f26e5ed1b930018941d348fd012d"} Oct 30 21:56:53.142880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.142860 4526 generic.go:356] "Generic (PLEG): container finished" podID="fb576fcd-2113-4ce7-92b7-852040b4aa1e" containerID="bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9" exitCode=0 Oct 30 21:56:53.143647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.143633 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" Oct 30 21:56:53.143773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.143758 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" event={"ID":"fb576fcd-2113-4ce7-92b7-852040b4aa1e","Type":"ContainerDied","Data":"bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9"} Oct 30 21:56:53.143836 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.143807 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs" event={"ID":"fb576fcd-2113-4ce7-92b7-852040b4aa1e","Type":"ContainerDied","Data":"9ae0de1fb0d53092d8dfa68c419780e62c3c071082e5b80bfe7294675d7f9d14"} Oct 30 21:56:53.143884 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.143839 4526 scope.go:122] "RemoveContainer" containerID="bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9" Oct 30 21:56:53.158052 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.158025 4526 scope.go:122] "RemoveContainer" containerID="bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9" Oct 30 21:56:53.158455 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:56:53.158380 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9\": container with ID starting with bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9 not found: ID does not exist" containerID="bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9" Oct 30 21:56:53.158455 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.158413 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9"} err="failed to get container status \"bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9\": rpc error: code = NotFound desc = could not find container \"bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9\": container with ID starting with bb31e598fab569d44816160ec4b114d19a1fb74ff18ab5f72b2fea0f0f5f33e9 not found: ID does not exist" Oct 30 21:56:53.232206 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.232140 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:56:53.248154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.248074 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb576fcd-2113-4ce7-92b7-852040b4aa1e-kube-api-access-sdk9b\" (UniqueName: \"kubernetes.io/projected/fb576fcd-2113-4ce7-92b7-852040b4aa1e-kube-api-access-sdk9b\") pod \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\" (UID: \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\") " Oct 30 21:56:53.248316 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.248289 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb576fcd-2113-4ce7-92b7-852040b4aa1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb576fcd-2113-4ce7-92b7-852040b4aa1e-plugins-config-volume\") pod \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\" (UID: \"fb576fcd-2113-4ce7-92b7-852040b4aa1e\") " Oct 30 21:56:53.249177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.248720 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb576fcd-2113-4ce7-92b7-852040b4aa1e-plugins-config-volume" pod "fb576fcd-2113-4ce7-92b7-852040b4aa1e" (UID: "fb576fcd-2113-4ce7-92b7-852040b4aa1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:53.249177 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.249163 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb576fcd-2113-4ce7-92b7-852040b4aa1e-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:53.251891 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.251864 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb576fcd-2113-4ce7-92b7-852040b4aa1e-kube-api-access-sdk9b" pod "fb576fcd-2113-4ce7-92b7-852040b4aa1e" (UID: "fb576fcd-2113-4ce7-92b7-852040b4aa1e"). InnerVolumeSpecName "kube-api-access-sdk9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:53.349644 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.349605 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-plugins-config-volume\") pod \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\" (UID: \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\") " Oct 30 21:56:53.349851 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.349750 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-kube-api-access-8nrbk\" (UniqueName: \"kubernetes.io/projected/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-kube-api-access-8nrbk\") pod \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\" (UID: \"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2\") " Oct 30 21:56:53.349982 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.349953 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-plugins-config-volume" pod "e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" (UID: "e1df4872-bfd8-40a7-9cf1-ca7acf1216f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:53.350110 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.350096 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:53.350155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.350114 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdk9b\" (UniqueName: \"kubernetes.io/projected/fb576fcd-2113-4ce7-92b7-852040b4aa1e-kube-api-access-sdk9b\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:53.351977 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.351954 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-kube-api-access-8nrbk" pod "e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" (UID: "e1df4872-bfd8-40a7-9cf1-ca7acf1216f2"). InnerVolumeSpecName "kube-api-access-8nrbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:53.357168 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.357112 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" podStartSLOduration=2.357096173 podStartE2EDuration="2.357096173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:53.353751624 +0000 UTC m=+2403.487124823" watchObservedRunningTime="2026-10-30 21:56:53.357096173 +0000 UTC m=+2403.490469373" Oct 30 21:56:53.415514 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:56:53.394326 4526 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda833c9ee_cf5b_4a50_bece_7d4e9be9d096.slice/crio-conmon-13eef02290b1ac82a1d817d773964c1c34b490b326afee98d806a25841120b89.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda833c9ee_cf5b_4a50_bece_7d4e9be9d096.slice/crio-13eef02290b1ac82a1d817d773964c1c34b490b326afee98d806a25841120b89.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:56:53.417057 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.416975 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:56:53.434718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.434681 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:56:53.434922 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.434743 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 21:56:53.451377 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.451342 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nrbk\" (UniqueName: \"kubernetes.io/projected/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2-kube-api-access-8nrbk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:53.620754 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.620698 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" podStartSLOduration=2.620681529 podStartE2EDuration="2.620681529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:53.593439528 +0000 UTC m=+2403.726812726" watchObservedRunningTime="2026-10-30 21:56:53.620681529 +0000 UTC m=+2403.754054730" Oct 30 21:56:53.787266 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.787225 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs"] Oct 30 21:56:53.818602 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.818550 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e4dadd-549899bd9c-9gqcs"] Oct 30 21:56:53.922398 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.922295 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr"] Oct 30 21:56:53.925343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.925317 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9796f1-zwfrr"] Oct 30 21:56:53.985889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.985853 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da690afa-ea30-4227-a106-c590b5a626ac" path="/var/lib/kubelet/pods/da690afa-ea30-4227-a106-c590b5a626ac/volumes" Oct 30 21:56:53.986249 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:53.986236 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb576fcd-2113-4ce7-92b7-852040b4aa1e" path="/var/lib/kubelet/pods/fb576fcd-2113-4ce7-92b7-852040b4aa1e/volumes" Oct 30 21:56:54.152041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.152007 4526 generic.go:356] "Generic (PLEG): container finished" podID="7e377970-46c6-4a2c-ac24-eb4f14116b4a" containerID="8cbc2430a598fff32ad44d9bb05c1c14b177e6e706d088ac0931dbfc450508af" exitCode=0 Oct 30 21:56:54.152468 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.152079 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" event={"ID":"7e377970-46c6-4a2c-ac24-eb4f14116b4a","Type":"ContainerDied","Data":"8cbc2430a598fff32ad44d9bb05c1c14b177e6e706d088ac0931dbfc450508af"} Oct 30 21:56:54.153764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.153736 4526 generic.go:356] "Generic (PLEG): container finished" podID="a833c9ee-cf5b-4a50-bece-7d4e9be9d096" containerID="13eef02290b1ac82a1d817d773964c1c34b490b326afee98d806a25841120b89" exitCode=0 Oct 30 21:56:54.153880 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.153776 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" event={"ID":"a833c9ee-cf5b-4a50-bece-7d4e9be9d096","Type":"ContainerDied","Data":"13eef02290b1ac82a1d817d773964c1c34b490b326afee98d806a25841120b89"} Oct 30 21:56:54.155111 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.155096 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" Oct 30 21:56:54.155193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.155120 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d" event={"ID":"e1df4872-bfd8-40a7-9cf1-ca7acf1216f2","Type":"ContainerDied","Data":"a35dde6815d98535bfa0a70b08fb584cee26f5e47e6019e1024e4005633bf7b9"} Oct 30 21:56:54.155193 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.155173 4526 scope.go:122] "RemoveContainer" containerID="b31041897174bef0f9f069e55cf9f9f388c6f26e5ed1b930018941d348fd012d" Oct 30 21:56:54.435061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.435026 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:56:54.435254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.435099 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 21:56:54.522736 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.522687 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d"] Oct 30 21:56:54.525759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:54.525721 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e4dadd-5c9548cc55-kmp5d"] Oct 30 21:56:55.377702 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.377676 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:55.426903 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.426844 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:55.434017 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.433995 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:56:55.434140 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.434036 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 21:56:55.475576 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.475545 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-openshift-ca\") pod \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " Oct 30 21:56:55.475725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.475582 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-xks-ca\") pod \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " Oct 30 21:56:55.475725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.475629 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e377970-46c6-4a2c-ac24-eb4f14116b4a-kube-api-access-mn7zk\" (UniqueName: \"kubernetes.io/projected/7e377970-46c6-4a2c-ac24-eb4f14116b4a-kube-api-access-mn7zk\") pod \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\" (UID: \"7e377970-46c6-4a2c-ac24-eb4f14116b4a\") " Oct 30 21:56:55.475725 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.475690 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-kube-api-access-f77kb\" (UniqueName: \"kubernetes.io/projected/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-kube-api-access-f77kb\") pod \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " Oct 30 21:56:55.475968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.475772 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-openshift-ca\") pod \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " Oct 30 21:56:55.475968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.475805 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-xks-ca\") pod \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\" (UID: \"a833c9ee-cf5b-4a50-bece-7d4e9be9d096\") " Oct 30 21:56:55.476142 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.476122 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-openshift-ca" pod "a833c9ee-cf5b-4a50-bece-7d4e9be9d096" (UID: "a833c9ee-cf5b-4a50-bece-7d4e9be9d096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:55.476406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.476346 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-openshift-ca" pod "7e377970-46c6-4a2c-ac24-eb4f14116b4a" (UID: "7e377970-46c6-4a2c-ac24-eb4f14116b4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:55.476406 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.476373 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:55.478173 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.478145 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-xks-ca" pod "7e377970-46c6-4a2c-ac24-eb4f14116b4a" (UID: "7e377970-46c6-4a2c-ac24-eb4f14116b4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:55.478301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.478196 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-kube-api-access-f77kb" pod "a833c9ee-cf5b-4a50-bece-7d4e9be9d096" (UID: "a833c9ee-cf5b-4a50-bece-7d4e9be9d096"). InnerVolumeSpecName "kube-api-access-f77kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:55.478624 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.478606 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e377970-46c6-4a2c-ac24-eb4f14116b4a-kube-api-access-mn7zk" pod "7e377970-46c6-4a2c-ac24-eb4f14116b4a" (UID: "7e377970-46c6-4a2c-ac24-eb4f14116b4a"). InnerVolumeSpecName "kube-api-access-mn7zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:55.478708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.478690 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-xks-ca" pod "a833c9ee-cf5b-4a50-bece-7d4e9be9d096" (UID: "a833c9ee-cf5b-4a50-bece-7d4e9be9d096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:55.577796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.577763 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:55.577796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.577789 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:55.577796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.577799 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e377970-46c6-4a2c-ac24-eb4f14116b4a-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:55.578034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.577811 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn7zk\" (UniqueName: \"kubernetes.io/projected/7e377970-46c6-4a2c-ac24-eb4f14116b4a-kube-api-access-mn7zk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:55.578034 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.577820 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f77kb\" (UniqueName: \"kubernetes.io/projected/a833c9ee-cf5b-4a50-bece-7d4e9be9d096-kube-api-access-f77kb\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:55.986145 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:55.986111 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" path="/var/lib/kubelet/pods/e1df4872-bfd8-40a7-9cf1-ca7acf1216f2/volumes" Oct 30 21:56:56.172732 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:56.172705 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" Oct 30 21:56:56.172912 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:56.172704 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2" event={"ID":"7e377970-46c6-4a2c-ac24-eb4f14116b4a","Type":"ContainerDied","Data":"cef00493c44370d3401249a8fe14a824bd2d79492e7891dc37b8e720dcbdaa4f"} Oct 30 21:56:56.172912 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:56.172856 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef00493c44370d3401249a8fe14a824bd2d79492e7891dc37b8e720dcbdaa4f" Oct 30 21:56:56.174209 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:56.174192 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" Oct 30 21:56:56.174312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:56.174272 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht" event={"ID":"a833c9ee-cf5b-4a50-bece-7d4e9be9d096","Type":"ContainerDied","Data":"7e1b6632ce1713e754ea907fd7dca9d832ec754cdcd89719ee59e1bf1a16e515"} Oct 30 21:56:56.174312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:56.174309 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e1b6632ce1713e754ea907fd7dca9d832ec754cdcd89719ee59e1bf1a16e515" Oct 30 21:56:56.434183 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:56.434146 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:56:56.434662 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:56.434220 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 21:56:57.438757 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:57.438726 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:56:57.440022 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:57.440002 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 21:56:58.279534 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.279387 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" containerID="cri-o://f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67" gracePeriod=45 Oct 30 21:56:58.594019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.593980 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:56:58.710993 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.710943 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tls-certs\") pod \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " Oct 30 21:56:58.711886 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.711850 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-model-cache\") pod \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " Oct 30 21:56:58.712041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.711940 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-home\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-home\") pod \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " Oct 30 21:56:58.712125 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.712097 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba054402-cbd1-4b77-8016-8bb37af9a9ae-kube-api-access-zht4v\" (UniqueName: \"kubernetes.io/projected/ba054402-cbd1-4b77-8016-8bb37af9a9ae-kube-api-access-zht4v\") pod \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " Oct 30 21:56:58.712200 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.712161 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-model-cache" pod "ba054402-cbd1-4b77-8016-8bb37af9a9ae" (UID: "ba054402-cbd1-4b77-8016-8bb37af9a9ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:58.712373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.712355 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-dshm\") pod \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " Oct 30 21:56:58.712458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.712373 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-home" pod "ba054402-cbd1-4b77-8016-8bb37af9a9ae" (UID: "ba054402-cbd1-4b77-8016-8bb37af9a9ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:58.712458 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.712400 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tmp-dir\") pod \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\" (UID: \"ba054402-cbd1-4b77-8016-8bb37af9a9ae\") " Oct 30 21:56:58.716575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.713106 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.716575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.713134 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.716575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.715400 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tmp-dir" pod "ba054402-cbd1-4b77-8016-8bb37af9a9ae" (UID: "ba054402-cbd1-4b77-8016-8bb37af9a9ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:58.716575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.716431 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tls-certs" pod "ba054402-cbd1-4b77-8016-8bb37af9a9ae" (UID: "ba054402-cbd1-4b77-8016-8bb37af9a9ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:58.716575 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.716539 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba054402-cbd1-4b77-8016-8bb37af9a9ae-kube-api-access-zht4v" pod "ba054402-cbd1-4b77-8016-8bb37af9a9ae" (UID: "ba054402-cbd1-4b77-8016-8bb37af9a9ae"). InnerVolumeSpecName "kube-api-access-zht4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:58.717583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.717560 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-dshm" pod "ba054402-cbd1-4b77-8016-8bb37af9a9ae" (UID: "ba054402-cbd1-4b77-8016-8bb37af9a9ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:58.814203 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.814169 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.814203 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.814200 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zht4v\" (UniqueName: \"kubernetes.io/projected/ba054402-cbd1-4b77-8016-8bb37af9a9ae-kube-api-access-zht4v\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.814203 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.814212 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.814560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:58.814242 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba054402-cbd1-4b77-8016-8bb37af9a9ae-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:59.194837 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.194808 4526 generic.go:356] "Generic (PLEG): container finished" podID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerID="f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67" exitCode=0 Oct 30 21:56:59.195045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.194856 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" Oct 30 21:56:59.195045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.194894 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" event={"ID":"ba054402-cbd1-4b77-8016-8bb37af9a9ae","Type":"ContainerDied","Data":"f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67"} Oct 30 21:56:59.195045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.194942 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg" event={"ID":"ba054402-cbd1-4b77-8016-8bb37af9a9ae","Type":"ContainerDied","Data":"90845f2f64cc0008cda32e781694303b5d9683e18d1310aba0d9b09aadc97c0e"} Oct 30 21:56:59.195045 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.194962 4526 scope.go:122] "RemoveContainer" containerID="f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67" Oct 30 21:56:59.205681 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.205654 4526 scope.go:122] "RemoveContainer" containerID="f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67" Oct 30 21:56:59.205995 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:56:59.205967 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67\": container with ID starting with f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67 not found: ID does not exist" containerID="f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67" Oct 30 21:56:59.206074 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.205996 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67"} err="failed to get container status \"f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67\": rpc error: code = NotFound desc = could not find container \"f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67\": container with ID starting with f56e4f99c9f9fbeb9075f9c2a353f5731c67f3909b89adce2e195090fe5cde67 not found: ID does not exist" Oct 30 21:56:59.358178 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.358147 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg"] Oct 30 21:56:59.406826 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.406790 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-021907/distinct-w5-021907-kserve-785c74b5cc-zrvxg"] Oct 30 21:56:59.985617 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:56:59.985583 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" path="/var/lib/kubelet/pods/ba054402-cbd1-4b77-8016-8bb37af9a9ae/volumes" Oct 30 21:57:00.430093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:00.430049 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg"] Oct 30 21:57:00.479301 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:00.479265 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b42c9-22vpg"] Oct 30 21:57:01.986095 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:01.986062 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdd914b-1e72-4d32-9a34-40e3ecb1b63d" path="/var/lib/kubelet/pods/9cdd914b-1e72-4d32-9a34-40e3ecb1b63d/volumes" Oct 30 21:57:04.875408 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.875370 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp"] Oct 30 21:57:04.875966 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.875948 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fb576fcd-2113-4ce7-92b7-852040b4aa1e" containerName="payload-processing" Oct 30 21:57:04.876027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.875967 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb576fcd-2113-4ce7-92b7-852040b4aa1e" containerName="payload-processing" Oct 30 21:57:04.876027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.875982 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a833c9ee-cf5b-4a50-bece-7d4e9be9d096" containerName="revoke-keys" Oct 30 21:57:04.876027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.875988 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a833c9ee-cf5b-4a50-bece-7d4e9be9d096" containerName="revoke-keys" Oct 30 21:57:04.876027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876004 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7e377970-46c6-4a2c-ac24-eb4f14116b4a" containerName="revoke-keys" Oct 30 21:57:04.876027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876009 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e377970-46c6-4a2c-ac24-eb4f14116b4a" containerName="revoke-keys" Oct 30 21:57:04.876027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876023 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" containerName="payload-pre-processing" Oct 30 21:57:04.876027 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876029 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" containerName="payload-pre-processing" Oct 30 21:57:04.876247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876037 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" Oct 30 21:57:04.876247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876042 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" Oct 30 21:57:04.876247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876112 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb576fcd-2113-4ce7-92b7-852040b4aa1e" containerName="payload-processing" Oct 30 21:57:04.876247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876122 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a833c9ee-cf5b-4a50-bece-7d4e9be9d096" containerName="revoke-keys" Oct 30 21:57:04.876247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876128 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e377970-46c6-4a2c-ac24-eb4f14116b4a" containerName="revoke-keys" Oct 30 21:57:04.876247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876135 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1df4872-bfd8-40a7-9cf1-ca7acf1216f2" containerName="payload-pre-processing" Oct 30 21:57:04.876247 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.876140 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba054402-cbd1-4b77-8016-8bb37af9a9ae" containerName="main" Oct 30 21:57:04.880804 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.880787 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:04.926056 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.926020 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z"] Oct 30 21:57:04.930023 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.930003 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:04.937338 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.937313 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp"] Oct 30 21:57:04.972312 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.972280 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:04.982429 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.982397 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3223b35-b76b-4a50-8334-876682df6579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:04.982612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.982455 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:04.982612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.982558 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbn6l\" (UniqueName: \"kubernetes.io/projected/c3223b35-b76b-4a50-8334-876682df6579-kube-api-access-cbn6l\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:04.982612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.982591 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3223b35-b76b-4a50-8334-876682df6579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:04.982730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.982675 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:57:04.982730 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.982705 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:04.982797 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.982743 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6mf\" (UniqueName: \"kubernetes.io/projected/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-kube-api-access-hl6mf\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:04.988146 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:04.988120 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z"] Oct 30 21:57:05.083921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.083888 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:05.084135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.083947 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbn6l\" (UniqueName: \"kubernetes.io/projected/c3223b35-b76b-4a50-8334-876682df6579-kube-api-access-cbn6l\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:05.084135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.083972 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3223b35-b76b-4a50-8334-876682df6579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:05.084135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.084082 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:05.084135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.084131 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6mf\" (UniqueName: \"kubernetes.io/projected/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-kube-api-access-hl6mf\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:05.084372 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.084204 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3223b35-b76b-4a50-8334-876682df6579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:05.084798 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.084774 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3223b35-b76b-4a50-8334-876682df6579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:05.084888 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.084831 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:05.086608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.086585 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:05.086705 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.086657 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3223b35-b76b-4a50-8334-876682df6579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:05.140780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.140708 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6mf\" (UniqueName: \"kubernetes.io/projected/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-kube-api-access-hl6mf\") pod \"maas-api-revoke-sub-f08cd6e2-8vplp\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:05.140921 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.140802 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbn6l\" (UniqueName: \"kubernetes.io/projected/c3223b35-b76b-4a50-8334-876682df6579-kube-api-access-cbn6l\") pod \"maas-api-revoke-sub-da0efe0b-bpp9z\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:05.194265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.194230 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:05.245244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.245213 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:05.578002 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.577973 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp"] Oct 30 21:57:05.578983 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:57:05.578954 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81e71d9_8c5d_4813_a741_bc2bb1b23a60.slice/crio-c853d8e025bca13a751c6d5157b50f4296bc119440148f62cd088d3427e5c045 WatchSource:0}: Error finding container c853d8e025bca13a751c6d5157b50f4296bc119440148f62cd088d3427e5c045: Status 404 returned error can't find the container with id c853d8e025bca13a751c6d5157b50f4296bc119440148f62cd088d3427e5c045 Oct 30 21:57:05.679996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:05.679974 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z"] Oct 30 21:57:05.681853 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:57:05.681829 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3223b35_b76b_4a50_8334_876682df6579.slice/crio-b4a387c1deacb343559b010dd413ccb912c525214923552d093b846bcea3bd53 WatchSource:0}: Error finding container b4a387c1deacb343559b010dd413ccb912c525214923552d093b846bcea3bd53: Status 404 returned error can't find the container with id b4a387c1deacb343559b010dd413ccb912c525214923552d093b846bcea3bd53 Oct 30 21:57:06.246358 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:06.246272 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" event={"ID":"a81e71d9-8c5d-4813-a741-bc2bb1b23a60","Type":"ContainerStarted","Data":"b6ab07c63baff310e4415fa38b4ae07cde443964afd6bbfa8d388a93e269251e"} Oct 30 21:57:06.246358 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:06.246338 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" event={"ID":"a81e71d9-8c5d-4813-a741-bc2bb1b23a60","Type":"ContainerStarted","Data":"c853d8e025bca13a751c6d5157b50f4296bc119440148f62cd088d3427e5c045"} Oct 30 21:57:06.248780 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:06.248753 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" event={"ID":"c3223b35-b76b-4a50-8334-876682df6579","Type":"ContainerStarted","Data":"3f9d2c003da9722d149d35d71b6953cf88ff805e18ca9ba8dfba298cac1728c7"} Oct 30 21:57:06.248906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:06.248798 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" event={"ID":"c3223b35-b76b-4a50-8334-876682df6579","Type":"ContainerStarted","Data":"b4a387c1deacb343559b010dd413ccb912c525214923552d093b846bcea3bd53"} Oct 30 21:57:06.360787 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:06.360732 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" podStartSLOduration=2.360715513 podStartE2EDuration="2.360715513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:06.357817938 +0000 UTC m=+2416.491191137" watchObservedRunningTime="2026-10-30 21:57:06.360715513 +0000 UTC m=+2416.494088711" Oct 30 21:57:06.472531 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:06.472466 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" podStartSLOduration=2.472448845 podStartE2EDuration="2.472448845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:06.469665218 +0000 UTC m=+2416.603038416" watchObservedRunningTime="2026-10-30 21:57:06.472448845 +0000 UTC m=+2416.605822038" Oct 30 21:57:07.260310 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:07.260268 4526 generic.go:356] "Generic (PLEG): container finished" podID="c3223b35-b76b-4a50-8334-876682df6579" containerID="3f9d2c003da9722d149d35d71b6953cf88ff805e18ca9ba8dfba298cac1728c7" exitCode=0 Oct 30 21:57:07.260896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:07.260358 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" event={"ID":"c3223b35-b76b-4a50-8334-876682df6579","Type":"ContainerDied","Data":"3f9d2c003da9722d149d35d71b6953cf88ff805e18ca9ba8dfba298cac1728c7"} Oct 30 21:57:07.262119 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:07.262097 4526 generic.go:356] "Generic (PLEG): container finished" podID="a81e71d9-8c5d-4813-a741-bc2bb1b23a60" containerID="b6ab07c63baff310e4415fa38b4ae07cde443964afd6bbfa8d388a93e269251e" exitCode=0 Oct 30 21:57:07.262241 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:07.262133 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" event={"ID":"a81e71d9-8c5d-4813-a741-bc2bb1b23a60","Type":"ContainerDied","Data":"b6ab07c63baff310e4415fa38b4ae07cde443964afd6bbfa8d388a93e269251e"} Oct 30 21:57:08.465349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.465324 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:08.521085 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.521058 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:08.529228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.529163 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-openshift-ca\") pod \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " Oct 30 21:57:08.529366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.529282 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-kube-api-access-hl6mf\" (UniqueName: \"kubernetes.io/projected/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-kube-api-access-hl6mf\") pod \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " Oct 30 21:57:08.529366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.529317 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-xks-ca\") pod \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\" (UID: \"a81e71d9-8c5d-4813-a741-bc2bb1b23a60\") " Oct 30 21:57:08.529591 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.529563 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-openshift-ca" pod "a81e71d9-8c5d-4813-a741-bc2bb1b23a60" (UID: "a81e71d9-8c5d-4813-a741-bc2bb1b23a60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:08.529802 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.529780 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:08.531558 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.531536 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-xks-ca" pod "a81e71d9-8c5d-4813-a741-bc2bb1b23a60" (UID: "a81e71d9-8c5d-4813-a741-bc2bb1b23a60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:08.531709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.531688 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-kube-api-access-hl6mf" pod "a81e71d9-8c5d-4813-a741-bc2bb1b23a60" (UID: "a81e71d9-8c5d-4813-a741-bc2bb1b23a60"). InnerVolumeSpecName "kube-api-access-hl6mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:08.630201 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.630166 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3223b35-b76b-4a50-8334-876682df6579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3223b35-b76b-4a50-8334-876682df6579-maas-api-xks-ca\") pod \"c3223b35-b76b-4a50-8334-876682df6579\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " Oct 30 21:57:08.630402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.630268 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3223b35-b76b-4a50-8334-876682df6579-kube-api-access-cbn6l\" (UniqueName: \"kubernetes.io/projected/c3223b35-b76b-4a50-8334-876682df6579-kube-api-access-cbn6l\") pod \"c3223b35-b76b-4a50-8334-876682df6579\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " Oct 30 21:57:08.630472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.630415 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3223b35-b76b-4a50-8334-876682df6579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3223b35-b76b-4a50-8334-876682df6579-maas-api-openshift-ca\") pod \"c3223b35-b76b-4a50-8334-876682df6579\" (UID: \"c3223b35-b76b-4a50-8334-876682df6579\") " Oct 30 21:57:08.630790 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.630760 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3223b35-b76b-4a50-8334-876682df6579-maas-api-openshift-ca" pod "c3223b35-b76b-4a50-8334-876682df6579" (UID: "c3223b35-b76b-4a50-8334-876682df6579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:08.630907 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.630831 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl6mf\" (UniqueName: \"kubernetes.io/projected/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-kube-api-access-hl6mf\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:08.630907 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.630854 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81e71d9-8c5d-4813-a741-bc2bb1b23a60-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:08.632630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.632596 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3223b35-b76b-4a50-8334-876682df6579-maas-api-xks-ca" pod "c3223b35-b76b-4a50-8334-876682df6579" (UID: "c3223b35-b76b-4a50-8334-876682df6579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:08.632733 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.632660 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3223b35-b76b-4a50-8334-876682df6579-kube-api-access-cbn6l" pod "c3223b35-b76b-4a50-8334-876682df6579" (UID: "c3223b35-b76b-4a50-8334-876682df6579"). InnerVolumeSpecName "kube-api-access-cbn6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:08.732336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.732301 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3223b35-b76b-4a50-8334-876682df6579-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:08.732336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.732331 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3223b35-b76b-4a50-8334-876682df6579-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:08.732336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:08.732341 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbn6l\" (UniqueName: \"kubernetes.io/projected/c3223b35-b76b-4a50-8334-876682df6579-kube-api-access-cbn6l\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:09.285763 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:09.285733 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" Oct 30 21:57:09.285987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:09.285770 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp" event={"ID":"a81e71d9-8c5d-4813-a741-bc2bb1b23a60","Type":"ContainerDied","Data":"c853d8e025bca13a751c6d5157b50f4296bc119440148f62cd088d3427e5c045"} Oct 30 21:57:09.285987 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:09.285820 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c853d8e025bca13a751c6d5157b50f4296bc119440148f62cd088d3427e5c045" Oct 30 21:57:09.287339 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:09.287314 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" event={"ID":"c3223b35-b76b-4a50-8334-876682df6579","Type":"ContainerDied","Data":"b4a387c1deacb343559b010dd413ccb912c525214923552d093b846bcea3bd53"} Oct 30 21:57:09.287502 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:09.287350 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4a387c1deacb343559b010dd413ccb912c525214923552d093b846bcea3bd53" Oct 30 21:57:09.287502 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:09.287357 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z" Oct 30 21:57:12.479711 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:57:12.479676 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:57:18.380917 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.380872 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84"] Oct 30 21:57:18.381700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.381678 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c3223b35-b76b-4a50-8334-876682df6579" containerName="revoke-keys" Oct 30 21:57:18.381700 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.381700 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3223b35-b76b-4a50-8334-876682df6579" containerName="revoke-keys" Oct 30 21:57:18.381878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.381724 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a81e71d9-8c5d-4813-a741-bc2bb1b23a60" containerName="revoke-keys" Oct 30 21:57:18.381878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.381731 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81e71d9-8c5d-4813-a741-bc2bb1b23a60" containerName="revoke-keys" Oct 30 21:57:18.381878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.381853 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3223b35-b76b-4a50-8334-876682df6579" containerName="revoke-keys" Oct 30 21:57:18.381878 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.381868 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81e71d9-8c5d-4813-a741-bc2bb1b23a60" containerName="revoke-keys" Oct 30 21:57:18.385304 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.385282 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.438285 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.438198 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:18.442988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.442956 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84"] Oct 30 21:57:18.456874 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.456838 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5p26\" (UniqueName: \"kubernetes.io/projected/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-kube-api-access-s5p26\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.457064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.456966 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.457064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.457035 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.487984 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.487942 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:57:18.558709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.558672 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.558916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.558817 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5p26\" (UniqueName: \"kubernetes.io/projected/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-kube-api-access-s5p26\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.558916 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.558863 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.559554 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.559435 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.561329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.561305 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.615650 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.615618 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5p26\" (UniqueName: \"kubernetes.io/projected/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-kube-api-access-s5p26\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:18.698481 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:18.698395 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:19.076048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:19.076023 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84"] Oct 30 21:57:19.077464 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:57:19.077439 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2f47e3_7867_4eba_84a1_0eaa4b92976d.slice/crio-925d276396034fa06bf8abb305f46353ed1e08f0de619d7a0913818abd000ec0 WatchSource:0}: Error finding container 925d276396034fa06bf8abb305f46353ed1e08f0de619d7a0913818abd000ec0: Status 404 returned error can't find the container with id 925d276396034fa06bf8abb305f46353ed1e08f0de619d7a0913818abd000ec0 Oct 30 21:57:19.366432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:19.366331 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" event={"ID":"ee2f47e3-7867-4eba-84a1-0eaa4b92976d","Type":"ContainerStarted","Data":"925d276396034fa06bf8abb305f46353ed1e08f0de619d7a0913818abd000ec0"} Oct 30 21:57:20.375215 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:20.375174 4526 generic.go:356] "Generic (PLEG): container finished" podID="ee2f47e3-7867-4eba-84a1-0eaa4b92976d" containerID="ad957717421aff33112cb51f063e69326f50a4fa3fb19fc3604f8121ffc56bb2" exitCode=0 Oct 30 21:57:20.375714 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:20.375265 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" event={"ID":"ee2f47e3-7867-4eba-84a1-0eaa4b92976d","Type":"ContainerDied","Data":"ad957717421aff33112cb51f063e69326f50a4fa3fb19fc3604f8121ffc56bb2"} Oct 30 21:57:21.530405 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.530379 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:21.696337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.696260 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-openshift-ca\") pod \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " Oct 30 21:57:21.696337 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.696319 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-xks-ca\") pod \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " Oct 30 21:57:21.696605 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.696367 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-kube-api-access-s5p26\" (UniqueName: \"kubernetes.io/projected/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-kube-api-access-s5p26\") pod \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\" (UID: \"ee2f47e3-7867-4eba-84a1-0eaa4b92976d\") " Oct 30 21:57:21.696687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.696652 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-openshift-ca" pod "ee2f47e3-7867-4eba-84a1-0eaa4b92976d" (UID: "ee2f47e3-7867-4eba-84a1-0eaa4b92976d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:21.696782 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.696758 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:21.698604 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.698580 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-xks-ca" pod "ee2f47e3-7867-4eba-84a1-0eaa4b92976d" (UID: "ee2f47e3-7867-4eba-84a1-0eaa4b92976d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:21.698739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.698720 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-kube-api-access-s5p26" pod "ee2f47e3-7867-4eba-84a1-0eaa4b92976d" (UID: "ee2f47e3-7867-4eba-84a1-0eaa4b92976d"). InnerVolumeSpecName "kube-api-access-s5p26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:21.798432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.798408 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:21.798432 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:21.798431 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5p26\" (UniqueName: \"kubernetes.io/projected/ee2f47e3-7867-4eba-84a1-0eaa4b92976d-kube-api-access-s5p26\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:22.389160 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:22.389132 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" Oct 30 21:57:22.389345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:22.389170 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84" event={"ID":"ee2f47e3-7867-4eba-84a1-0eaa4b92976d","Type":"ContainerDied","Data":"925d276396034fa06bf8abb305f46353ed1e08f0de619d7a0913818abd000ec0"} Oct 30 21:57:22.389345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:22.389218 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925d276396034fa06bf8abb305f46353ed1e08f0de619d7a0913818abd000ec0" Oct 30 21:57:29.861404 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:29.861362 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx"] Oct 30 21:57:29.971699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:29.971664 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6"] Oct 30 21:57:34.862342 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:34.862296 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" podUID="5b4439a8-9ca4-4f02-862e-7ec85e0f9561" containerName="payload-processing" containerID="cri-o://939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a" gracePeriod=25 Oct 30 21:57:34.972586 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:34.972540 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" podUID="d9fcaf4b-8e8e-458a-bac1-41bf91806289" containerName="payload-pre-processing" containerID="cri-o://b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df" gracePeriod=25 Oct 30 21:57:35.127150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.127083 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:57:35.227997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.227975 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:57:35.288631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.288592 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-plugins-config-volume\") pod \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\" (UID: \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\") " Oct 30 21:57:35.288823 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.288754 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-kube-api-access-6qzkb\" (UniqueName: \"kubernetes.io/projected/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-kube-api-access-6qzkb\") pod \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\" (UID: \"5b4439a8-9ca4-4f02-862e-7ec85e0f9561\") " Oct 30 21:57:35.289007 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.288984 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-plugins-config-volume" pod "5b4439a8-9ca4-4f02-862e-7ec85e0f9561" (UID: "5b4439a8-9ca4-4f02-862e-7ec85e0f9561"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:35.289170 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.289153 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:35.290989 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.290969 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-kube-api-access-6qzkb" pod "5b4439a8-9ca4-4f02-862e-7ec85e0f9561" (UID: "5b4439a8-9ca4-4f02-862e-7ec85e0f9561"). InnerVolumeSpecName "kube-api-access-6qzkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:35.389718 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.389635 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9fcaf4b-8e8e-458a-bac1-41bf91806289-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcaf4b-8e8e-458a-bac1-41bf91806289-plugins-config-volume\") pod \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\" (UID: \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\") " Oct 30 21:57:35.389919 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.389897 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9fcaf4b-8e8e-458a-bac1-41bf91806289-kube-api-access-4m484\" (UniqueName: \"kubernetes.io/projected/d9fcaf4b-8e8e-458a-bac1-41bf91806289-kube-api-access-4m484\") pod \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\" (UID: \"d9fcaf4b-8e8e-458a-bac1-41bf91806289\") " Oct 30 21:57:35.390066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.390042 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fcaf4b-8e8e-458a-bac1-41bf91806289-plugins-config-volume" pod "d9fcaf4b-8e8e-458a-bac1-41bf91806289" (UID: "d9fcaf4b-8e8e-458a-bac1-41bf91806289"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:35.390343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.390316 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcaf4b-8e8e-458a-bac1-41bf91806289-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:35.390343 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.390335 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzkb\" (UniqueName: \"kubernetes.io/projected/5b4439a8-9ca4-4f02-862e-7ec85e0f9561-kube-api-access-6qzkb\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:35.392349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.392327 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fcaf4b-8e8e-458a-bac1-41bf91806289-kube-api-access-4m484" pod "d9fcaf4b-8e8e-458a-bac1-41bf91806289" (UID: "d9fcaf4b-8e8e-458a-bac1-41bf91806289"). InnerVolumeSpecName "kube-api-access-4m484". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:35.487417 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.487385 4526 generic.go:356] "Generic (PLEG): container finished" podID="5b4439a8-9ca4-4f02-862e-7ec85e0f9561" containerID="939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a" exitCode=0 Oct 30 21:57:35.487653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.487429 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" Oct 30 21:57:35.487653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.487469 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" event={"ID":"5b4439a8-9ca4-4f02-862e-7ec85e0f9561","Type":"ContainerDied","Data":"939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a"} Oct 30 21:57:35.487653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.487541 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx" event={"ID":"5b4439a8-9ca4-4f02-862e-7ec85e0f9561","Type":"ContainerDied","Data":"0b73efee691d6351ebd0052b297e7e92a402eaf81165a721b03a622039d04306"} Oct 30 21:57:35.487653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.487569 4526 scope.go:122] "RemoveContainer" containerID="939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a" Oct 30 21:57:35.489064 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.489045 4526 generic.go:356] "Generic (PLEG): container finished" podID="d9fcaf4b-8e8e-458a-bac1-41bf91806289" containerID="b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df" exitCode=0 Oct 30 21:57:35.489149 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.489086 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" Oct 30 21:57:35.489198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.489137 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" event={"ID":"d9fcaf4b-8e8e-458a-bac1-41bf91806289","Type":"ContainerDied","Data":"b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df"} Oct 30 21:57:35.489198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.489190 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6" event={"ID":"d9fcaf4b-8e8e-458a-bac1-41bf91806289","Type":"ContainerDied","Data":"768f5c620d94c09104ccd22e3cc99254caf83b35c3702da36b16e94536a30765"} Oct 30 21:57:35.490986 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.490964 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m484\" (UniqueName: \"kubernetes.io/projected/d9fcaf4b-8e8e-458a-bac1-41bf91806289-kube-api-access-4m484\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:35.506262 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.506180 4526 scope.go:122] "RemoveContainer" containerID="939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a" Oct 30 21:57:35.506740 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:57:35.506709 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a\": container with ID starting with 939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a not found: ID does not exist" containerID="939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a" Oct 30 21:57:35.506859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.506753 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a"} err="failed to get container status \"939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a\": rpc error: code = NotFound desc = could not find container \"939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a\": container with ID starting with 939f2e1be5dfa34955fd7ae031f99ac952116f05efd0086623e0f56ddf780d8a not found: ID does not exist" Oct 30 21:57:35.506859 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.506778 4526 scope.go:122] "RemoveContainer" containerID="b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df" Oct 30 21:57:35.518413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.518394 4526 scope.go:122] "RemoveContainer" containerID="b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df" Oct 30 21:57:35.518688 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:57:35.518668 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df\": container with ID starting with b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df not found: ID does not exist" containerID="b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df" Oct 30 21:57:35.518772 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.518700 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df"} err="failed to get container status \"b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df\": rpc error: code = NotFound desc = could not find container \"b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df\": container with ID starting with b70100ce086a2240547e4eb5b179d8439b55edc8fc963d05d0c6267586fc57df not found: ID does not exist" Oct 30 21:57:35.651412 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.651315 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6"] Oct 30 21:57:35.654174 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.654151 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-3b21b4-df55bf4b7-8xxf6"] Oct 30 21:57:35.761212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.761178 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx"] Oct 30 21:57:35.765225 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.765193 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-3b21b4-667967d7d5-24pkx"] Oct 30 21:57:35.985653 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.985580 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b4439a8-9ca4-4f02-862e-7ec85e0f9561" path="/var/lib/kubelet/pods/5b4439a8-9ca4-4f02-862e-7ec85e0f9561/volumes" Oct 30 21:57:35.986019 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:35.985978 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9fcaf4b-8e8e-458a-bac1-41bf91806289" path="/var/lib/kubelet/pods/d9fcaf4b-8e8e-458a-bac1-41bf91806289/volumes" Oct 30 21:57:36.401336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401303 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4"] Oct 30 21:57:36.401831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401817 4526 cpu_manager.go:414] "containerMap: removing container" podUID="d9fcaf4b-8e8e-458a-bac1-41bf91806289" containerName="payload-pre-processing" Oct 30 21:57:36.401892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401832 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9fcaf4b-8e8e-458a-bac1-41bf91806289" containerName="payload-pre-processing" Oct 30 21:57:36.401892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401861 4526 cpu_manager.go:414] "containerMap: removing container" podUID="5b4439a8-9ca4-4f02-862e-7ec85e0f9561" containerName="payload-processing" Oct 30 21:57:36.401892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401868 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b4439a8-9ca4-4f02-862e-7ec85e0f9561" containerName="payload-processing" Oct 30 21:57:36.401892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401877 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ee2f47e3-7867-4eba-84a1-0eaa4b92976d" containerName="revoke-keys" Oct 30 21:57:36.401892 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401882 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2f47e3-7867-4eba-84a1-0eaa4b92976d" containerName="revoke-keys" Oct 30 21:57:36.402055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401973 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9fcaf4b-8e8e-458a-bac1-41bf91806289" containerName="payload-pre-processing" Oct 30 21:57:36.402055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401986 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b4439a8-9ca4-4f02-862e-7ec85e0f9561" containerName="payload-processing" Oct 30 21:57:36.402055 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.401994 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2f47e3-7867-4eba-84a1-0eaa4b92976d" containerName="revoke-keys" Oct 30 21:57:36.406721 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.406702 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.460220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.460190 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-75c2acf6\"/\"kube-root-ca.crt\"" Oct 30 21:57:36.460595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.460368 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-75c2acf6\"/\"default-dockercfg-v4ltn\"" Oct 30 21:57:36.460595 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.460455 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-75c2acf6\"/\"openshift-service-ca.crt\"" Oct 30 21:57:36.465255 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.465227 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4"] Oct 30 21:57:36.502321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.502285 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-dshm\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.502524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.502340 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tls-certs\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.502524 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.502467 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-home\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.502630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.502583 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tmp-dir\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.502630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.502620 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv4zz\" (UniqueName: \"kubernetes.io/projected/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-kube-api-access-dv4zz\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.502715 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.502670 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-model-cache\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.509413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.509394 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-75c2acf6\"/\"e2e-lifecycle-model-75c2acf6-kserve-self-signed-certs\"" Oct 30 21:57:36.603939 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.603906 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-dshm\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.604148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.603950 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tls-certs\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.604148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.603997 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-home\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.604148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.604045 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tmp-dir\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.604148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.604071 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv4zz\" (UniqueName: \"kubernetes.io/projected/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-kube-api-access-dv4zz\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.604148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.604096 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-model-cache\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.604470 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.604443 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-home\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.604564 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.604474 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-model-cache\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.604616 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.604569 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tmp-dir\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.606512 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.606463 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-dshm\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.606825 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.606804 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tls-certs\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.660642 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.660572 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv4zz\" (UniqueName: \"kubernetes.io/projected/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-kube-api-access-dv4zz\") pod \"e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:36.719391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:36.719356 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:37.100071 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:37.100042 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4"] Oct 30 21:57:37.101738 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:57:37.101707 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fe0ccf4_b9cc_4948_92fd_5f2fedc7176f.slice/crio-e8ab04cacbd93b871a0c0f25a4c21102e3ffa15eb5ca5ed7690c04d626fef81a WatchSource:0}: Error finding container e8ab04cacbd93b871a0c0f25a4c21102e3ffa15eb5ca5ed7690c04d626fef81a: Status 404 returned error can't find the container with id e8ab04cacbd93b871a0c0f25a4c21102e3ffa15eb5ca5ed7690c04d626fef81a Oct 30 21:57:37.508583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:37.508449 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" event={"ID":"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f","Type":"ContainerStarted","Data":"29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6"} Oct 30 21:57:37.508583 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:37.508524 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" event={"ID":"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f","Type":"ContainerStarted","Data":"e8ab04cacbd93b871a0c0f25a4c21102e3ffa15eb5ca5ed7690c04d626fef81a"} Oct 30 21:57:37.625761 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:37.625695 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" podStartSLOduration=1.625674657 podStartE2EDuration="1.625674657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:37.623061992 +0000 UTC m=+2447.756435192" watchObservedRunningTime="2026-10-30 21:57:37.625674657 +0000 UTC m=+2447.759047856" Oct 30 21:57:37.720053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:37.720021 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:42.477780 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:57:42.477745 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:57:42.669784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:42.669755 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58"] Oct 30 21:57:42.719220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:42.719191 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-021907-wzs58"] Oct 30 21:57:43.912378 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:43.912347 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84"] Oct 30 21:57:43.961555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:43.961522 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b21b4-99r84"] Oct 30 21:57:43.986265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:43.986232 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8b1499-f78b-417c-b8cf-e85918b8f227" path="/var/lib/kubelet/pods/8f8b1499-f78b-417c-b8cf-e85918b8f227/volumes" Oct 30 21:57:43.986694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:43.986681 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee2f47e3-7867-4eba-84a1-0eaa4b92976d" path="/var/lib/kubelet/pods/ee2f47e3-7867-4eba-84a1-0eaa4b92976d/volumes" Oct 30 21:57:46.720413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:46.720382 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:46.733729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:46.733703 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:47.587605 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:47.587573 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 21:57:49.476346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:49.476312 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm"] Oct 30 21:57:49.476786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:49.476526 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" containerID="cri-o://4e852ba86c135a80be8188e11d50857a5900e2c930a3f452edfe844a9f9c5408" gracePeriod=30 Oct 30 21:57:50.450630 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:50.450599 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz"] Oct 30 21:57:50.519833 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:50.519790 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e4dadd-s2dtz"] Oct 30 21:57:51.984620 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:51.984573 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b803d94d-e17b-4514-bdd8-8b2066f01eca" path="/var/lib/kubelet/pods/b803d94d-e17b-4514-bdd8-8b2066f01eca/volumes" Oct 30 21:57:52.143805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:52.143771 4526 scope.go:122] "RemoveContainer" containerID="429ccc8d74990369bdbada2b4796d39e0d8f8b4d83644cf580dbf8d93303223f" Oct 30 21:57:52.154694 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:52.154671 4526 scope.go:122] "RemoveContainer" containerID="4d675e347adbdcb5b00870bf253b9e5fc73ff1b4fcf5307591d69079516fb09e" Oct 30 21:57:52.164203 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:52.164175 4526 scope.go:122] "RemoveContainer" containerID="4b6b7ce9e3f24201d5d2ba6e19b742e2adce041b153de2a6427d694d072aef20" Oct 30 21:57:54.653329 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.653285 4526 generic.go:356] "Generic (PLEG): container finished" podID="46444220-52f6-4122-b602-6a2a6531a4de" containerID="4e852ba86c135a80be8188e11d50857a5900e2c930a3f452edfe844a9f9c5408" exitCode=0 Oct 30 21:57:54.653850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.653368 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" event={"ID":"46444220-52f6-4122-b602-6a2a6531a4de","Type":"ContainerDied","Data":"4e852ba86c135a80be8188e11d50857a5900e2c930a3f452edfe844a9f9c5408"} Oct 30 21:57:54.653850 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.653522 4526 scope.go:122] "RemoveContainer" containerID="4aecfe60cd248537425ce239a0062ca1a0c1ae43814424e8b120108aca8274c4" Oct 30 21:57:54.769557 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.769529 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:57:54.853612 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.853576 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-kube-api-access-27pdv\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-kube-api-access-27pdv\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.853803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.853664 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-istio-token\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-istio-token\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.853803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.853774 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-certs\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.853926 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.853833 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-credential-socket\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.853988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.853929 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46444220-52f6-4122-b602-6a2a6531a4de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46444220-52f6-4122-b602-6a2a6531a4de-istio-podinfo\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.853988 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.853963 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-data\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.854097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854022 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-socket\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.854097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854020 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-certs" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.854097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854089 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-envoy\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.854265 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854147 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46444220-52f6-4122-b602-6a2a6531a4de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46444220-52f6-4122-b602-6a2a6531a4de-istiod-ca-cert\") pod \"46444220-52f6-4122-b602-6a2a6531a4de\" (UID: \"46444220-52f6-4122-b602-6a2a6531a4de\") " Oct 30 21:57:54.854328 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854088 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-credential-socket" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.854328 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854231 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-data" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.854328 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854295 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-socket" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.854685 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854664 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.854764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854692 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.854764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854707 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.854764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854655 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46444220-52f6-4122-b602-6a2a6531a4de-istiod-ca-cert" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:54.854929 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.854751 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.856415 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.856385 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46444220-52f6-4122-b602-6a2a6531a4de-istio-podinfo" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:54.856699 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.856678 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-kube-api-access-27pdv" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "kube-api-access-27pdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:54.856830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.856750 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-envoy" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.856830 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.856808 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-istio-token" pod "46444220-52f6-4122-b602-6a2a6531a4de" (UID: "46444220-52f6-4122-b602-6a2a6531a4de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:54.955773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.955733 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46444220-52f6-4122-b602-6a2a6531a4de-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.955773 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.955773 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46444220-52f6-4122-b602-6a2a6531a4de-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.955971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.955787 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46444220-52f6-4122-b602-6a2a6531a4de-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.955971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.955801 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27pdv\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-kube-api-access-27pdv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.955971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:54.955815 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46444220-52f6-4122-b602-6a2a6531a4de-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:55.662175 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:55.662139 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" event={"ID":"46444220-52f6-4122-b602-6a2a6531a4de","Type":"ContainerDied","Data":"72f80181d0a56abed91c560f9a40e1e79ef8243c6ccc210e170cc266dca19206"} Oct 30 21:57:55.662657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:55.662206 4526 scope.go:122] "RemoveContainer" containerID="4e852ba86c135a80be8188e11d50857a5900e2c930a3f452edfe844a9f9c5408" Oct 30 21:57:55.662657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:55.662148 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm" Oct 30 21:57:55.875703 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:55.875654 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm"] Oct 30 21:57:55.878757 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:55.878730 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-g8wtm"] Oct 30 21:57:55.989211 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:55.987251 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46444220-52f6-4122-b602-6a2a6531a4de" path="/var/lib/kubelet/pods/46444220-52f6-4122-b602-6a2a6531a4de/volumes" Oct 30 21:57:56.672728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:56.672648 4526 generic.go:356] "Generic (PLEG): container finished" podID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerID="fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d" exitCode=0 Oct 30 21:57:56.673272 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:56.672785 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" event={"ID":"351cddf1-72d9-404d-b1e8-d1592a4bffe8","Type":"ContainerDied","Data":"fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d"} Oct 30 21:57:56.673626 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:56.673603 4526 scope.go:122] "RemoveContainer" containerID="fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d" Oct 30 21:57:56.683185 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:56.683150 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:56.683331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:56.683230 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:56.683331 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:56.683262 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:57.681471 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:57.681435 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" event={"ID":"351cddf1-72d9-404d-b1e8-d1592a4bffe8","Type":"ContainerStarted","Data":"d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22"} Oct 30 21:57:57.760050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:57.760015 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:57:57.760253 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:57.760208 4526 patch_prober.go:29] interesting pod/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 21:57:57.760308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:57.760249 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 21:57:58.760899 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:58.760868 4526 patch_prober.go:29] interesting pod/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 21:57:58.761286 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:58.760919 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 21:57:59.760306 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:59.760269 4526 patch_prober.go:29] interesting pod/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 21:57:59.760530 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:57:59.760334 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 21:58:00.760770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:00.760727 4526 patch_prober.go:29] interesting pod/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 21:58:00.761299 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:00.760794 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 21:58:01.760577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:01.760474 4526 patch_prober.go:29] interesting pod/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 21:58:01.760577 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:01.760539 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 21:58:02.760516 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:02.760447 4526 patch_prober.go:29] interesting pod/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 21:58:02.760978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:02.760568 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 21:58:03.765192 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:03.765162 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:58:03.765680 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:03.765471 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:58:03.766228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:03.766210 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 21:58:12.032537 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.032505 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55"] Oct 30 21:58:12.033041 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.033027 4526 cpu_manager.go:414] "containerMap: removing container" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" Oct 30 21:58:12.033106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.033043 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" Oct 30 21:58:12.033106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.033060 4526 cpu_manager.go:414] "containerMap: removing container" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" Oct 30 21:58:12.033106 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.033065 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" Oct 30 21:58:12.033207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.033163 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" Oct 30 21:58:12.033207 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.033172 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="46444220-52f6-4122-b602-6a2a6531a4de" containerName="istio-proxy" Oct 30 21:58:12.037371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.037356 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.093796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.093768 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55"] Oct 30 21:58:12.139260 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.139225 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:12.139452 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.139281 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:58:12.199242 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.199210 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885d213-56eb-44ef-8391-253a91e64983-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.199442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.199251 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885d213-56eb-44ef-8391-253a91e64983-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.199531 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.199457 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pg9\" (UniqueName: \"kubernetes.io/projected/d885d213-56eb-44ef-8391-253a91e64983-kube-api-access-j5pg9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.300387 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.300346 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pg9\" (UniqueName: \"kubernetes.io/projected/d885d213-56eb-44ef-8391-253a91e64983-kube-api-access-j5pg9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.300643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.300436 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885d213-56eb-44ef-8391-253a91e64983-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.300643 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.300463 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885d213-56eb-44ef-8391-253a91e64983-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.301198 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.301174 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885d213-56eb-44ef-8391-253a91e64983-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.303121 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.303099 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885d213-56eb-44ef-8391-253a91e64983-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.357656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.357622 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pg9\" (UniqueName: \"kubernetes.io/projected/d885d213-56eb-44ef-8391-253a91e64983-kube-api-access-j5pg9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:12.478854 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:58:12.478806 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:58:12.651590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:12.651502 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:13.029308 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:13.029277 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55"] Oct 30 21:58:13.031079 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:58:13.031040 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd885d213_56eb_44ef_8391_253a91e64983.slice/crio-744aa3cec6e52b2b4dc1a1acc31ebd44ee3b58236cf30857229cb1f6083d865e WatchSource:0}: Error finding container 744aa3cec6e52b2b4dc1a1acc31ebd44ee3b58236cf30857229cb1f6083d865e: Status 404 returned error can't find the container with id 744aa3cec6e52b2b4dc1a1acc31ebd44ee3b58236cf30857229cb1f6083d865e Oct 30 21:58:13.807572 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:13.807536 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" event={"ID":"d885d213-56eb-44ef-8391-253a91e64983","Type":"ContainerStarted","Data":"6fceeeeaa3e147a26630a2f40657c67204dfce1b8d11aea38c343f70fedfc302"} Oct 30 21:58:13.807978 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:13.807599 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" event={"ID":"d885d213-56eb-44ef-8391-253a91e64983","Type":"ContainerStarted","Data":"744aa3cec6e52b2b4dc1a1acc31ebd44ee3b58236cf30857229cb1f6083d865e"} Oct 30 21:58:13.922442 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:13.922384 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" podStartSLOduration=2.922371267 podStartE2EDuration="2.922371267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:13.919312267 +0000 UTC m=+2484.052685464" watchObservedRunningTime="2026-10-30 21:58:13.922371267 +0000 UTC m=+2484.055744505" Oct 30 21:58:14.815770 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:14.815736 4526 generic.go:356] "Generic (PLEG): container finished" podID="d885d213-56eb-44ef-8391-253a91e64983" containerID="6fceeeeaa3e147a26630a2f40657c67204dfce1b8d11aea38c343f70fedfc302" exitCode=0 Oct 30 21:58:14.816213 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:14.815824 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" event={"ID":"d885d213-56eb-44ef-8391-253a91e64983","Type":"ContainerDied","Data":"6fceeeeaa3e147a26630a2f40657c67204dfce1b8d11aea38c343f70fedfc302"} Oct 30 21:58:15.959300 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:15.959278 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:16.036475 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.036436 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d885d213-56eb-44ef-8391-253a91e64983-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885d213-56eb-44ef-8391-253a91e64983-maas-api-xks-ca\") pod \"d885d213-56eb-44ef-8391-253a91e64983\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " Oct 30 21:58:16.036724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.036650 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d885d213-56eb-44ef-8391-253a91e64983-kube-api-access-j5pg9\" (UniqueName: \"kubernetes.io/projected/d885d213-56eb-44ef-8391-253a91e64983-kube-api-access-j5pg9\") pod \"d885d213-56eb-44ef-8391-253a91e64983\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " Oct 30 21:58:16.036724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.036698 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d885d213-56eb-44ef-8391-253a91e64983-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885d213-56eb-44ef-8391-253a91e64983-maas-api-openshift-ca\") pod \"d885d213-56eb-44ef-8391-253a91e64983\" (UID: \"d885d213-56eb-44ef-8391-253a91e64983\") " Oct 30 21:58:16.037141 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.037111 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d885d213-56eb-44ef-8391-253a91e64983-maas-api-openshift-ca" pod "d885d213-56eb-44ef-8391-253a91e64983" (UID: "d885d213-56eb-44ef-8391-253a91e64983"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:16.038860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.038839 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d885d213-56eb-44ef-8391-253a91e64983-maas-api-xks-ca" pod "d885d213-56eb-44ef-8391-253a91e64983" (UID: "d885d213-56eb-44ef-8391-253a91e64983"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:16.038860 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.038852 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d885d213-56eb-44ef-8391-253a91e64983-kube-api-access-j5pg9" pod "d885d213-56eb-44ef-8391-253a91e64983" (UID: "d885d213-56eb-44ef-8391-253a91e64983"). InnerVolumeSpecName "kube-api-access-j5pg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:16.137845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.137755 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885d213-56eb-44ef-8391-253a91e64983-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:16.137845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.137788 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pg9\" (UniqueName: \"kubernetes.io/projected/d885d213-56eb-44ef-8391-253a91e64983-kube-api-access-j5pg9\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:16.137845 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.137800 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885d213-56eb-44ef-8391-253a91e64983-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:16.831912 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.831870 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" Oct 30 21:58:16.832097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.831932 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55" event={"ID":"d885d213-56eb-44ef-8391-253a91e64983","Type":"ContainerDied","Data":"744aa3cec6e52b2b4dc1a1acc31ebd44ee3b58236cf30857229cb1f6083d865e"} Oct 30 21:58:16.832097 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:16.832010 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="744aa3cec6e52b2b4dc1a1acc31ebd44ee3b58236cf30857229cb1f6083d865e" Oct 30 21:58:24.315303 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:24.315265 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj"] Oct 30 21:58:24.422189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:24.422152 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr"] Oct 30 21:58:29.316375 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.316325 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" podUID="b47e4ad1-21dc-448c-b169-6230907371e9" containerName="payload-processing" containerID="cri-o://1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113" gracePeriod=25 Oct 30 21:58:29.422542 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.422458 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" podUID="44a5707e-b59f-44f5-b934-03be6da8edea" containerName="payload-pre-processing" containerID="cri-o://da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70" gracePeriod=25 Oct 30 21:58:29.610077 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.610044 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:58:29.695276 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.695244 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:58:29.698333 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.698309 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47e4ad1-21dc-448c-b169-6230907371e9-kube-api-access-8lnpv\" (UniqueName: \"kubernetes.io/projected/b47e4ad1-21dc-448c-b169-6230907371e9-kube-api-access-8lnpv\") pod \"b47e4ad1-21dc-448c-b169-6230907371e9\" (UID: \"b47e4ad1-21dc-448c-b169-6230907371e9\") " Oct 30 21:58:29.698573 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.698560 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b47e4ad1-21dc-448c-b169-6230907371e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47e4ad1-21dc-448c-b169-6230907371e9-plugins-config-volume\") pod \"b47e4ad1-21dc-448c-b169-6230907371e9\" (UID: \"b47e4ad1-21dc-448c-b169-6230907371e9\") " Oct 30 21:58:29.698908 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.698881 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47e4ad1-21dc-448c-b169-6230907371e9-plugins-config-volume" pod "b47e4ad1-21dc-448c-b169-6230907371e9" (UID: "b47e4ad1-21dc-448c-b169-6230907371e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:29.700652 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.700633 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47e4ad1-21dc-448c-b169-6230907371e9-kube-api-access-8lnpv" pod "b47e4ad1-21dc-448c-b169-6230907371e9" (UID: "b47e4ad1-21dc-448c-b169-6230907371e9"). InnerVolumeSpecName "kube-api-access-8lnpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:29.800244 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.800198 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a5707e-b59f-44f5-b934-03be6da8edea-kube-api-access-bh5fs\" (UniqueName: \"kubernetes.io/projected/44a5707e-b59f-44f5-b934-03be6da8edea-kube-api-access-bh5fs\") pod \"44a5707e-b59f-44f5-b934-03be6da8edea\" (UID: \"44a5707e-b59f-44f5-b934-03be6da8edea\") " Oct 30 21:58:29.800433 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.800410 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a5707e-b59f-44f5-b934-03be6da8edea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a5707e-b59f-44f5-b934-03be6da8edea-plugins-config-volume\") pod \"44a5707e-b59f-44f5-b934-03be6da8edea\" (UID: \"44a5707e-b59f-44f5-b934-03be6da8edea\") " Oct 30 21:58:29.800787 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.800770 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47e4ad1-21dc-448c-b169-6230907371e9-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:29.800787 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.800788 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lnpv\" (UniqueName: \"kubernetes.io/projected/b47e4ad1-21dc-448c-b169-6230907371e9-kube-api-access-8lnpv\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:29.800923 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.800793 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a5707e-b59f-44f5-b934-03be6da8edea-plugins-config-volume" pod "44a5707e-b59f-44f5-b934-03be6da8edea" (UID: "44a5707e-b59f-44f5-b934-03be6da8edea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:29.802609 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.802582 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a5707e-b59f-44f5-b934-03be6da8edea-kube-api-access-bh5fs" pod "44a5707e-b59f-44f5-b934-03be6da8edea" (UID: "44a5707e-b59f-44f5-b934-03be6da8edea"). InnerVolumeSpecName "kube-api-access-bh5fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:29.902288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.902192 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a5707e-b59f-44f5-b934-03be6da8edea-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:29.902288 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.902226 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh5fs\" (UniqueName: \"kubernetes.io/projected/44a5707e-b59f-44f5-b934-03be6da8edea-kube-api-access-bh5fs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:29.915472 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.915437 4526 generic.go:356] "Generic (PLEG): container finished" podID="44a5707e-b59f-44f5-b934-03be6da8edea" containerID="da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70" exitCode=0 Oct 30 21:58:29.915717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.915510 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" Oct 30 21:58:29.915717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.915535 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" event={"ID":"44a5707e-b59f-44f5-b934-03be6da8edea","Type":"ContainerDied","Data":"da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70"} Oct 30 21:58:29.915717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.915596 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr" event={"ID":"44a5707e-b59f-44f5-b934-03be6da8edea","Type":"ContainerDied","Data":"17d23ce425335a57b6122acfae3f29b76ffcac3af4223d83a2cf181e38fb9ecd"} Oct 30 21:58:29.915717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.915625 4526 scope.go:122] "RemoveContainer" containerID="da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70" Oct 30 21:58:29.917050 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.917007 4526 generic.go:356] "Generic (PLEG): container finished" podID="b47e4ad1-21dc-448c-b169-6230907371e9" containerID="1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113" exitCode=0 Oct 30 21:58:29.917202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.917048 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" event={"ID":"b47e4ad1-21dc-448c-b169-6230907371e9","Type":"ContainerDied","Data":"1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113"} Oct 30 21:58:29.917202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.917075 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" Oct 30 21:58:29.917202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.917100 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj" event={"ID":"b47e4ad1-21dc-448c-b169-6230907371e9","Type":"ContainerDied","Data":"7b7e68428b07c0ed20f83bd7183fe92fdb9342892871ff81a37f826c0368ab0e"} Oct 30 21:58:29.927450 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.927421 4526 scope.go:122] "RemoveContainer" containerID="da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70" Oct 30 21:58:29.928243 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:58:29.928156 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70\": container with ID starting with da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70 not found: ID does not exist" containerID="da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70" Oct 30 21:58:29.928370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.928252 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70"} err="failed to get container status \"da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70\": rpc error: code = NotFound desc = could not find container \"da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70\": container with ID starting with da558bbc8d8bb8b32430f989b3e380ee75efe68adeb36ba52dc20c1c22c53e70 not found: ID does not exist" Oct 30 21:58:29.928370 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.928330 4526 scope.go:122] "RemoveContainer" containerID="1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113" Oct 30 21:58:29.944340 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.944309 4526 scope.go:122] "RemoveContainer" containerID="1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113" Oct 30 21:58:29.944702 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:58:29.944677 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113\": container with ID starting with 1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113 not found: ID does not exist" containerID="1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113" Oct 30 21:58:29.944786 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:29.944712 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113"} err="failed to get container status \"1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113\": rpc error: code = NotFound desc = could not find container \"1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113\": container with ID starting with 1de9d89f0e746d7147e72155846c5400ffd6e76c4b1bdd8a43dd6accc26fd113 not found: ID does not exist" Oct 30 21:58:30.081667 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:30.081631 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr"] Oct 30 21:58:30.085307 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:30.085277 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6a241-96b78b6c9-q8frr"] Oct 30 21:58:30.193762 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:30.193664 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj"] Oct 30 21:58:30.198113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:30.198082 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6a241-867b98b47-f68xj"] Oct 30 21:58:31.984202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:31.984160 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a5707e-b59f-44f5-b934-03be6da8edea" path="/var/lib/kubelet/pods/44a5707e-b59f-44f5-b934-03be6da8edea/volumes" Oct 30 21:58:31.984865 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:31.984802 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47e4ad1-21dc-448c-b169-6230907371e9" path="/var/lib/kubelet/pods/b47e4ad1-21dc-448c-b169-6230907371e9/volumes" Oct 30 21:58:42.479600 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:58:42.479560 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:58:52.254345 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:52.254305 4526 scope.go:122] "RemoveContainer" containerID="35f4ee97d810edfab9a50051f5dc1ad55459f9f04d1efdfe607e592753379a54" Oct 30 21:58:52.264156 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:52.264129 4526 scope.go:122] "RemoveContainer" containerID="3275c78f8a605e1490220976c5587e438ac9875bdebaafa31a34311d9c36f734" Oct 30 21:58:53.567426 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:53.567344 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55"] Oct 30 21:58:53.616672 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:53.616635 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6a241-dlg55"] Oct 30 21:58:53.985724 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:58:53.985647 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d885d213-56eb-44ef-8391-253a91e64983" path="/var/lib/kubelet/pods/d885d213-56eb-44ef-8391-253a91e64983/volumes" Oct 30 21:59:01.716161 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716122 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b"] Oct 30 21:59:01.716670 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716655 4526 cpu_manager.go:414] "containerMap: removing container" podUID="b47e4ad1-21dc-448c-b169-6230907371e9" containerName="payload-processing" Oct 30 21:59:01.716729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716671 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47e4ad1-21dc-448c-b169-6230907371e9" containerName="payload-processing" Oct 30 21:59:01.716729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716684 4526 cpu_manager.go:414] "containerMap: removing container" podUID="44a5707e-b59f-44f5-b934-03be6da8edea" containerName="payload-pre-processing" Oct 30 21:59:01.716729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716689 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a5707e-b59f-44f5-b934-03be6da8edea" containerName="payload-pre-processing" Oct 30 21:59:01.716729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716703 4526 cpu_manager.go:414] "containerMap: removing container" podUID="d885d213-56eb-44ef-8391-253a91e64983" containerName="revoke-keys" Oct 30 21:59:01.716729 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716708 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d885d213-56eb-44ef-8391-253a91e64983" containerName="revoke-keys" Oct 30 21:59:01.716920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716795 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47e4ad1-21dc-448c-b169-6230907371e9" containerName="payload-processing" Oct 30 21:59:01.716920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716802 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a5707e-b59f-44f5-b934-03be6da8edea" containerName="payload-pre-processing" Oct 30 21:59:01.716920 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.716809 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="d885d213-56eb-44ef-8391-253a91e64983" containerName="revoke-keys" Oct 30 21:59:01.721712 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.721695 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:01.777231 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.777196 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b"] Oct 30 21:59:01.822747 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.822714 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:01.822948 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.822826 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:59:01.855670 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.855632 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbdc\" (UniqueName: \"kubernetes.io/projected/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-kube-api-access-grbdc\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:01.855854 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.855745 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:01.855910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.855892 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:01.956759 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.956719 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:01.956960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.956779 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbdc\" (UniqueName: \"kubernetes.io/projected/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-kube-api-access-grbdc\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:01.956960 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.956823 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:01.957548 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.957523 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:01.959346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:01.959326 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:02.014774 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:02.014696 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbdc\" (UniqueName: \"kubernetes.io/projected/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-kube-api-access-grbdc\") pod \"maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:02.035605 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:02.035553 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:02.414517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:02.414464 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b"] Oct 30 21:59:02.415427 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:59:02.415401 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01d6b98_0a0b_446f_8a9a_c7d4e569fb7b.slice/crio-8f1b4c81027c42bfabb04ecdeaa873b3f025f0ef772df978426c17363b133e15 WatchSource:0}: Error finding container 8f1b4c81027c42bfabb04ecdeaa873b3f025f0ef772df978426c17363b133e15: Status 404 returned error can't find the container with id 8f1b4c81027c42bfabb04ecdeaa873b3f025f0ef772df978426c17363b133e15 Oct 30 21:59:03.148346 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:03.148304 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" event={"ID":"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b","Type":"ContainerStarted","Data":"cd67142fc7924a63958c4e59fdfe5f69b2ac5f799e2014b59b2a75cc44d5dec2"} Oct 30 21:59:03.148910 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:03.148404 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" event={"ID":"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b","Type":"ContainerStarted","Data":"8f1b4c81027c42bfabb04ecdeaa873b3f025f0ef772df978426c17363b133e15"} Oct 30 21:59:03.265635 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:03.265579 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" podStartSLOduration=2.265562293 podStartE2EDuration="2.265562293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:03.261971229 +0000 UTC m=+2533.395344428" watchObservedRunningTime="2026-10-30 21:59:03.265562293 +0000 UTC m=+2533.398935491" Oct 30 21:59:04.155708 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:04.155670 4526 generic.go:356] "Generic (PLEG): container finished" podID="c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b" containerID="cd67142fc7924a63958c4e59fdfe5f69b2ac5f799e2014b59b2a75cc44d5dec2" exitCode=0 Oct 30 21:59:04.156133 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:04.155745 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" event={"ID":"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b","Type":"ContainerDied","Data":"cd67142fc7924a63958c4e59fdfe5f69b2ac5f799e2014b59b2a75cc44d5dec2"} Oct 30 21:59:05.304945 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.304920 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:05.395995 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.395962 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-kube-api-access-grbdc\" (UniqueName: \"kubernetes.io/projected/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-kube-api-access-grbdc\") pod \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " Oct 30 21:59:05.396182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.396043 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-openshift-ca\") pod \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " Oct 30 21:59:05.396182 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.396072 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-xks-ca\") pod \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\" (UID: \"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b\") " Oct 30 21:59:05.396413 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.396391 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-openshift-ca" pod "c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b" (UID: "c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:05.398374 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.398342 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-kube-api-access-grbdc" pod "c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b" (UID: "c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b"). InnerVolumeSpecName "kube-api-access-grbdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:05.398526 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.398428 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-xks-ca" pod "c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b" (UID: "c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:05.497599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.497515 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbdc\" (UniqueName: \"kubernetes.io/projected/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-kube-api-access-grbdc\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.497599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.497551 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:05.497599 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:05.497566 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:06.138366 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:06.138328 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd"] Oct 30 21:59:06.172748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:06.172094 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" Oct 30 21:59:06.172748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:06.172104 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b" event={"ID":"c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b","Type":"ContainerDied","Data":"8f1b4c81027c42bfabb04ecdeaa873b3f025f0ef772df978426c17363b133e15"} Oct 30 21:59:06.172748 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:06.172156 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f1b4c81027c42bfabb04ecdeaa873b3f025f0ef772df978426c17363b133e15" Oct 30 21:59:12.481374 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:59:12.481337 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:59:12.660202 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.660164 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp"] Oct 30 21:59:12.660739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.660724 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b" containerName="revoke-keys" Oct 30 21:59:12.660803 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.660741 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b" containerName="revoke-keys" Oct 30 21:59:12.660841 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.660835 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b" containerName="revoke-keys" Oct 30 21:59:12.664501 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.664471 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.705212 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.705178 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6"] Oct 30 21:59:12.709021 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.709001 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:12.718590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.718566 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:12.751517 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.751432 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:59:12.754533 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.754481 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp"] Oct 30 21:59:12.771896 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.771863 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6"] Oct 30 21:59:12.782954 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.782876 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.783122 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.783076 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.783242 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.783218 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ghb\" (UniqueName: \"kubernetes.io/projected/ed76201b-10ad-4151-a8b0-ba70936b4480-kube-api-access-p7ghb\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.884093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.884052 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.884093 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.884097 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:12.884327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.884166 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ghb\" (UniqueName: \"kubernetes.io/projected/ed76201b-10ad-4151-a8b0-ba70936b4480-kube-api-access-p7ghb\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.884327 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.884296 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cjs\" (UniqueName: \"kubernetes.io/projected/64a33d5c-6993-41bc-a39f-66bd97452b18-kube-api-access-d7cjs\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:12.884425 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.884405 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.884483 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.884463 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:12.885048 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.885022 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.886717 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.886698 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.942889 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.942850 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ghb\" (UniqueName: \"kubernetes.io/projected/ed76201b-10ad-4151-a8b0-ba70936b4480-kube-api-access-p7ghb\") pod \"maas-api-revoke-sub-3661db54-p9dwp\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.977661 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.977628 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:12.985282 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.985258 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:12.985402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.985309 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cjs\" (UniqueName: \"kubernetes.io/projected/64a33d5c-6993-41bc-a39f-66bd97452b18-kube-api-access-d7cjs\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:12.985402 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.985366 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:12.986013 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.985987 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:12.988115 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:12.988096 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:13.066906 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:13.066848 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cjs\" (UniqueName: \"kubernetes.io/projected/64a33d5c-6993-41bc-a39f-66bd97452b18-kube-api-access-d7cjs\") pod \"maas-api-revoke-sub-d102b1b7-mmfb6\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:13.236590 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:13.236554 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt"] Oct 30 21:59:13.322099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:13.322071 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:13.369459 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:13.369426 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp"] Oct 30 21:59:13.406822 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:59:13.406791 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded76201b_10ad_4151_a8b0_ba70936b4480.slice/crio-03461d99a04394dc23a5cc951915eab6b24e9276b9208492989b6bdd1835658f WatchSource:0}: Error finding container 03461d99a04394dc23a5cc951915eab6b24e9276b9208492989b6bdd1835658f: Status 404 returned error can't find the container with id 03461d99a04394dc23a5cc951915eab6b24e9276b9208492989b6bdd1835658f Oct 30 21:59:13.714720 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:13.714691 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6"] Oct 30 21:59:13.716519 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:59:13.716449 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a33d5c_6993_41bc_a39f_66bd97452b18.slice/crio-d69997d66f670eb54f8743770b5a2e97634af8cf2ba8f70e123c8288118476be WatchSource:0}: Error finding container d69997d66f670eb54f8743770b5a2e97634af8cf2ba8f70e123c8288118476be: Status 404 returned error can't find the container with id d69997d66f670eb54f8743770b5a2e97634af8cf2ba8f70e123c8288118476be Oct 30 21:59:14.224194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:14.224099 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" event={"ID":"ed76201b-10ad-4151-a8b0-ba70936b4480","Type":"ContainerStarted","Data":"695fc4c237e0449d3ce26e9644d3652d4a7c3bff5853b82e04a5a42f370b47d8"} Oct 30 21:59:14.224194 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:14.224163 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" event={"ID":"ed76201b-10ad-4151-a8b0-ba70936b4480","Type":"ContainerStarted","Data":"03461d99a04394dc23a5cc951915eab6b24e9276b9208492989b6bdd1835658f"} Oct 30 21:59:14.225867 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:14.225840 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" event={"ID":"64a33d5c-6993-41bc-a39f-66bd97452b18","Type":"ContainerStarted","Data":"1d5d50697dd4a3ace694573a6688246f81ec0d763f511c83439b5343157debc0"} Oct 30 21:59:14.225996 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:14.225890 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" event={"ID":"64a33d5c-6993-41bc-a39f-66bd97452b18","Type":"ContainerStarted","Data":"d69997d66f670eb54f8743770b5a2e97634af8cf2ba8f70e123c8288118476be"} Oct 30 21:59:14.339155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:14.339107 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" podStartSLOduration=2.339090461 podStartE2EDuration="2.339090461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:14.33845856 +0000 UTC m=+2544.471831757" watchObservedRunningTime="2026-10-30 21:59:14.339090461 +0000 UTC m=+2544.472463661" Oct 30 21:59:14.460189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:14.460141 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" podStartSLOduration=2.46012708 podStartE2EDuration="2.46012708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:14.456263239 +0000 UTC m=+2544.589636437" watchObservedRunningTime="2026-10-30 21:59:14.46012708 +0000 UTC m=+2544.593500278" Oct 30 21:59:15.234188 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:15.234156 4526 generic.go:356] "Generic (PLEG): container finished" podID="ed76201b-10ad-4151-a8b0-ba70936b4480" containerID="695fc4c237e0449d3ce26e9644d3652d4a7c3bff5853b82e04a5a42f370b47d8" exitCode=0 Oct 30 21:59:15.234631 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:15.234242 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" event={"ID":"ed76201b-10ad-4151-a8b0-ba70936b4480","Type":"ContainerDied","Data":"695fc4c237e0449d3ce26e9644d3652d4a7c3bff5853b82e04a5a42f370b47d8"} Oct 30 21:59:15.236163 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:15.236139 4526 generic.go:356] "Generic (PLEG): container finished" podID="64a33d5c-6993-41bc-a39f-66bd97452b18" containerID="1d5d50697dd4a3ace694573a6688246f81ec0d763f511c83439b5343157debc0" exitCode=0 Oct 30 21:59:15.236313 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:15.236221 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" event={"ID":"64a33d5c-6993-41bc-a39f-66bd97452b18","Type":"ContainerDied","Data":"1d5d50697dd4a3ace694573a6688246f81ec0d763f511c83439b5343157debc0"} Oct 30 21:59:16.421687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.421661 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:16.425238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.425215 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:16.520373 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.520283 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-openshift-ca\") pod \"ed76201b-10ad-4151-a8b0-ba70936b4480\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " Oct 30 21:59:16.520608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.520401 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a33d5c-6993-41bc-a39f-66bd97452b18-kube-api-access-d7cjs\" (UniqueName: \"kubernetes.io/projected/64a33d5c-6993-41bc-a39f-66bd97452b18-kube-api-access-d7cjs\") pod \"64a33d5c-6993-41bc-a39f-66bd97452b18\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " Oct 30 21:59:16.520608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.520427 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-xks-ca\") pod \"ed76201b-10ad-4151-a8b0-ba70936b4480\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " Oct 30 21:59:16.520608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.520466 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-xks-ca\") pod \"64a33d5c-6993-41bc-a39f-66bd97452b18\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " Oct 30 21:59:16.520608 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.520546 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed76201b-10ad-4151-a8b0-ba70936b4480-kube-api-access-p7ghb\" (UniqueName: \"kubernetes.io/projected/ed76201b-10ad-4151-a8b0-ba70936b4480-kube-api-access-p7ghb\") pod \"ed76201b-10ad-4151-a8b0-ba70936b4480\" (UID: \"ed76201b-10ad-4151-a8b0-ba70936b4480\") " Oct 30 21:59:16.520863 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.520619 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-openshift-ca\") pod \"64a33d5c-6993-41bc-a39f-66bd97452b18\" (UID: \"64a33d5c-6993-41bc-a39f-66bd97452b18\") " Oct 30 21:59:16.520863 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.520718 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-openshift-ca" pod "ed76201b-10ad-4151-a8b0-ba70936b4480" (UID: "ed76201b-10ad-4151-a8b0-ba70936b4480"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:16.521066 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.521036 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-openshift-ca" pod "64a33d5c-6993-41bc-a39f-66bd97452b18" (UID: "64a33d5c-6993-41bc-a39f-66bd97452b18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:16.521180 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.521061 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:16.522904 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.522881 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a33d5c-6993-41bc-a39f-66bd97452b18-kube-api-access-d7cjs" pod "64a33d5c-6993-41bc-a39f-66bd97452b18" (UID: "64a33d5c-6993-41bc-a39f-66bd97452b18"). InnerVolumeSpecName "kube-api-access-d7cjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:16.523005 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.522937 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed76201b-10ad-4151-a8b0-ba70936b4480-kube-api-access-p7ghb" pod "ed76201b-10ad-4151-a8b0-ba70936b4480" (UID: "ed76201b-10ad-4151-a8b0-ba70936b4480"). InnerVolumeSpecName "kube-api-access-p7ghb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:16.523240 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.523220 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-xks-ca" pod "64a33d5c-6993-41bc-a39f-66bd97452b18" (UID: "64a33d5c-6993-41bc-a39f-66bd97452b18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:16.523321 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.523237 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-xks-ca" pod "ed76201b-10ad-4151-a8b0-ba70936b4480" (UID: "ed76201b-10ad-4151-a8b0-ba70936b4480"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:16.622274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.622239 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7cjs\" (UniqueName: \"kubernetes.io/projected/64a33d5c-6993-41bc-a39f-66bd97452b18-kube-api-access-d7cjs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:16.622274 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.622268 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed76201b-10ad-4151-a8b0-ba70936b4480-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:16.622593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.622302 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:16.622593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.622311 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7ghb\" (UniqueName: \"kubernetes.io/projected/ed76201b-10ad-4151-a8b0-ba70936b4480-kube-api-access-p7ghb\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:16.622593 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:16.622321 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a33d5c-6993-41bc-a39f-66bd97452b18-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:17.250570 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:17.250538 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" Oct 30 21:59:17.250768 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:17.250537 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp" event={"ID":"ed76201b-10ad-4151-a8b0-ba70936b4480","Type":"ContainerDied","Data":"03461d99a04394dc23a5cc951915eab6b24e9276b9208492989b6bdd1835658f"} Oct 30 21:59:17.250768 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:17.250674 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03461d99a04394dc23a5cc951915eab6b24e9276b9208492989b6bdd1835658f" Oct 30 21:59:17.252113 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:17.252093 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" Oct 30 21:59:17.252238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:17.252094 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6" event={"ID":"64a33d5c-6993-41bc-a39f-66bd97452b18","Type":"ContainerDied","Data":"d69997d66f670eb54f8743770b5a2e97634af8cf2ba8f70e123c8288118476be"} Oct 30 21:59:17.252238 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:17.252204 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d69997d66f670eb54f8743770b5a2e97634af8cf2ba8f70e123c8288118476be" Oct 30 21:59:18.237764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:18.237700 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" podUID="3d1c303c-e1ac-4e5e-bfd0-a66142a14770" containerName="payload-processing" containerID="cri-o://9ffcba5d88db4c0c850b5d2e5ab0d314de891a6ac478cf9f0951c577a6e849ea" gracePeriod=25 Oct 30 21:59:21.159368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.159327 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" containerID="cri-o://4cf1bad8af77ba3f2d61651848973d575b3c365b61a2694f077975de4e98c7d0" gracePeriod=45 Oct 30 21:59:21.283968 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.283933 4526 generic.go:356] "Generic (PLEG): container finished" podID="e631b930-be02-42a4-949a-a74e9b884a24" containerID="4cf1bad8af77ba3f2d61651848973d575b3c365b61a2694f077975de4e98c7d0" exitCode=0 Oct 30 21:59:21.284118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.284006 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" event={"ID":"e631b930-be02-42a4-949a-a74e9b884a24","Type":"ContainerDied","Data":"4cf1bad8af77ba3f2d61651848973d575b3c365b61a2694f077975de4e98c7d0"} Oct 30 21:59:21.426336 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.426278 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:59:21.479053 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479023 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-tmp-dir\") pod \"e631b930-be02-42a4-949a-a74e9b884a24\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " Oct 30 21:59:21.479220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479083 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-model-cache\") pod \"e631b930-be02-42a4-949a-a74e9b884a24\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " Oct 30 21:59:21.479220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479131 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-home\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-home\") pod \"e631b930-be02-42a4-949a-a74e9b884a24\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " Oct 30 21:59:21.479220 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479177 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-dshm\") pod \"e631b930-be02-42a4-949a-a74e9b884a24\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " Oct 30 21:59:21.479396 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479220 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e631b930-be02-42a4-949a-a74e9b884a24-tls-certs\" (UniqueName: \"kubernetes.io/secret/e631b930-be02-42a4-949a-a74e9b884a24-tls-certs\") pod \"e631b930-be02-42a4-949a-a74e9b884a24\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " Oct 30 21:59:21.479396 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479257 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e631b930-be02-42a4-949a-a74e9b884a24-kube-api-access-t4z88\" (UniqueName: \"kubernetes.io/projected/e631b930-be02-42a4-949a-a74e9b884a24-kube-api-access-t4z88\") pod \"e631b930-be02-42a4-949a-a74e9b884a24\" (UID: \"e631b930-be02-42a4-949a-a74e9b884a24\") " Oct 30 21:59:21.479396 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479328 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-tmp-dir" pod "e631b930-be02-42a4-949a-a74e9b884a24" (UID: "e631b930-be02-42a4-949a-a74e9b884a24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.479596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479372 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-model-cache" pod "e631b930-be02-42a4-949a-a74e9b884a24" (UID: "e631b930-be02-42a4-949a-a74e9b884a24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.479596 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479454 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-home" pod "e631b930-be02-42a4-949a-a74e9b884a24" (UID: "e631b930-be02-42a4-949a-a74e9b884a24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.479728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479677 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.479728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479698 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.479728 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.479713 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.481687 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.481654 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e631b930-be02-42a4-949a-a74e9b884a24-tls-certs" pod "e631b930-be02-42a4-949a-a74e9b884a24" (UID: "e631b930-be02-42a4-949a-a74e9b884a24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:21.481805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.481696 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-dshm" pod "e631b930-be02-42a4-949a-a74e9b884a24" (UID: "e631b930-be02-42a4-949a-a74e9b884a24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.481805 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.481744 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e631b930-be02-42a4-949a-a74e9b884a24-kube-api-access-t4z88" pod "e631b930-be02-42a4-949a-a74e9b884a24" (UID: "e631b930-be02-42a4-949a-a74e9b884a24"). InnerVolumeSpecName "kube-api-access-t4z88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:21.580990 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.580960 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e631b930-be02-42a4-949a-a74e9b884a24-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.580990 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.580988 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4z88\" (UniqueName: \"kubernetes.io/projected/e631b930-be02-42a4-949a-a74e9b884a24-kube-api-access-t4z88\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.581189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:21.580998 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e631b930-be02-42a4-949a-a74e9b884a24-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:22.291330 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:22.291299 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" Oct 30 21:59:22.291784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:22.291336 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd" event={"ID":"e631b930-be02-42a4-949a-a74e9b884a24","Type":"ContainerDied","Data":"dacdd44358c1e177d011613069ce556625634f4ea65a0fa3d420cd92494b368d"} Oct 30 21:59:22.291784 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:22.291390 4526 scope.go:122] "RemoveContainer" containerID="4cf1bad8af77ba3f2d61651848973d575b3c365b61a2694f077975de4e98c7d0" Oct 30 21:59:22.454038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:22.453996 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd"] Oct 30 21:59:22.503555 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:22.503522 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/premium-w0-1b5b5f-kserve-7b9f78b759-h4rrd"] Oct 30 21:59:23.985371 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:23.985335 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e631b930-be02-42a4-949a-a74e9b884a24" path="/var/lib/kubelet/pods/e631b930-be02-42a4-949a-a74e9b884a24/volumes" Oct 30 21:59:26.169971 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:26.169934 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" podUID="3d1c303c-e1ac-4e5e-bfd0-a66142a14770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:59:36.169533 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:36.169409 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" podUID="3d1c303c-e1ac-4e5e-bfd0-a66142a14770" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:59:37.867393 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.867359 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56"] Oct 30 21:59:37.867979 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.867963 4526 cpu_manager.go:414] "containerMap: removing container" podUID="64a33d5c-6993-41bc-a39f-66bd97452b18" containerName="revoke-keys" Oct 30 21:59:37.868030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.867981 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a33d5c-6993-41bc-a39f-66bd97452b18" containerName="revoke-keys" Oct 30 21:59:37.868030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.867996 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ed76201b-10ad-4151-a8b0-ba70936b4480" containerName="revoke-keys" Oct 30 21:59:37.868030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.868001 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed76201b-10ad-4151-a8b0-ba70936b4480" containerName="revoke-keys" Oct 30 21:59:37.868030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.868012 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" Oct 30 21:59:37.868030 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.868017 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" Oct 30 21:59:37.868189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.868091 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a33d5c-6993-41bc-a39f-66bd97452b18" containerName="revoke-keys" Oct 30 21:59:37.868189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.868101 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed76201b-10ad-4151-a8b0-ba70936b4480" containerName="revoke-keys" Oct 30 21:59:37.868189 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.868112 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e631b930-be02-42a4-949a-a74e9b884a24" containerName="main" Oct 30 21:59:37.872758 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.872740 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:37.930726 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.930681 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56"] Oct 30 21:59:37.971664 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.971629 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mskfz\" (UniqueName: \"kubernetes.io/projected/dcd74a57-2cf4-4826-ad00-e6657d2c2590-kube-api-access-mskfz\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:37.971835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.971669 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:37.971835 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.971771 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:37.976796 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.976780 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:37.976868 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:37.976808 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:59:38.072520 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.072468 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mskfz\" (UniqueName: \"kubernetes.io/projected/dcd74a57-2cf4-4826-ad00-e6657d2c2590-kube-api-access-mskfz\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:38.072738 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.072532 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:38.072738 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.072570 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:38.073277 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.073250 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:38.075063 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.075043 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:38.129449 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.129372 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mskfz\" (UniqueName: \"kubernetes.io/projected/dcd74a57-2cf4-4826-ad00-e6657d2c2590-kube-api-access-mskfz\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:38.186378 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.186342 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:38.406739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.406649 4526 generic.go:356] "Generic (PLEG): container finished" podID="3d1c303c-e1ac-4e5e-bfd0-a66142a14770" containerID="9ffcba5d88db4c0c850b5d2e5ab0d314de891a6ac478cf9f0951c577a6e849ea" exitCode=0 Oct 30 21:59:38.406739 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.406724 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" event={"ID":"3d1c303c-e1ac-4e5e-bfd0-a66142a14770","Type":"ContainerDied","Data":"9ffcba5d88db4c0c850b5d2e5ab0d314de891a6ac478cf9f0951c577a6e849ea"} Oct 30 21:59:38.488228 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.488203 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:59:38.565536 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.565507 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56"] Oct 30 21:59:38.570890 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:59:38.570829 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd74a57_2cf4_4826_ad00_e6657d2c2590.slice/crio-798295ff015b87b953a0ce3513a18e38ea93e4b8118062b86422c945a4fae558 WatchSource:0}: Error finding container 798295ff015b87b953a0ce3513a18e38ea93e4b8118062b86422c945a4fae558: Status 404 returned error can't find the container with id 798295ff015b87b953a0ce3513a18e38ea93e4b8118062b86422c945a4fae558 Oct 30 21:59:38.578118 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.578094 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-kube-api-access-n2pkt\" (UniqueName: \"kubernetes.io/projected/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-kube-api-access-n2pkt\") pod \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\" (UID: \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\") " Oct 30 21:59:38.578226 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.578127 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-plugins-config-volume\") pod \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\" (UID: \"3d1c303c-e1ac-4e5e-bfd0-a66142a14770\") " Oct 30 21:59:38.578560 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.578538 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-plugins-config-volume" pod "3d1c303c-e1ac-4e5e-bfd0-a66142a14770" (UID: "3d1c303c-e1ac-4e5e-bfd0-a66142a14770"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:38.580513 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.580470 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-kube-api-access-n2pkt" pod "3d1c303c-e1ac-4e5e-bfd0-a66142a14770" (UID: "3d1c303c-e1ac-4e5e-bfd0-a66142a14770"). InnerVolumeSpecName "kube-api-access-n2pkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:38.679657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.679569 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2pkt\" (UniqueName: \"kubernetes.io/projected/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-kube-api-access-n2pkt\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:38.679657 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:38.679601 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1c303c-e1ac-4e5e-bfd0-a66142a14770-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:39.415061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.414973 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" Oct 30 21:59:39.415061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.414985 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt" event={"ID":"3d1c303c-e1ac-4e5e-bfd0-a66142a14770","Type":"ContainerDied","Data":"28550eec84e710398a10e4442b02584bfe3430ac778c8553b12ecc8fb0094f88"} Oct 30 21:59:39.415061 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.415057 4526 scope.go:122] "RemoveContainer" containerID="9ffcba5d88db4c0c850b5d2e5ab0d314de891a6ac478cf9f0951c577a6e849ea" Oct 30 21:59:39.417120 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.417079 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" event={"ID":"dcd74a57-2cf4-4826-ad00-e6657d2c2590","Type":"ContainerStarted","Data":"464cafa32093cddea1b0fa3c8776c436bc40aee95019587cb11245cb5bbccc64"} Oct 30 21:59:39.417245 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.417128 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" event={"ID":"dcd74a57-2cf4-4826-ad00-e6657d2c2590","Type":"ContainerStarted","Data":"798295ff015b87b953a0ce3513a18e38ea93e4b8118062b86422c945a4fae558"} Oct 30 21:59:39.580155 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.580124 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt"] Oct 30 21:59:39.628275 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.628238 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1b5b5f-6766bd67c-mn6jt"] Oct 30 21:59:39.694656 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.694550 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" podStartSLOduration=2.694533806 podStartE2EDuration="2.694533806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:39.693759609 +0000 UTC m=+2569.827132808" watchObservedRunningTime="2026-10-30 21:59:39.694533806 +0000 UTC m=+2569.827907001" Oct 30 21:59:39.986297 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:39.986221 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1c303c-e1ac-4e5e-bfd0-a66142a14770" path="/var/lib/kubelet/pods/3d1c303c-e1ac-4e5e-bfd0-a66142a14770/volumes" Oct 30 21:59:40.424887 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:40.424854 4526 generic.go:356] "Generic (PLEG): container finished" podID="dcd74a57-2cf4-4826-ad00-e6657d2c2590" containerID="464cafa32093cddea1b0fa3c8776c436bc40aee95019587cb11245cb5bbccc64" exitCode=0 Oct 30 21:59:40.425391 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:40.424909 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" event={"ID":"dcd74a57-2cf4-4826-ad00-e6657d2c2590","Type":"ContainerDied","Data":"464cafa32093cddea1b0fa3c8776c436bc40aee95019587cb11245cb5bbccc64"} Oct 30 21:59:41.571289 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.571260 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:41.714263 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.714178 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd74a57-2cf4-4826-ad00-e6657d2c2590-kube-api-access-mskfz\" (UniqueName: \"kubernetes.io/projected/dcd74a57-2cf4-4826-ad00-e6657d2c2590-kube-api-access-mskfz\") pod \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " Oct 30 21:59:41.714263 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.714227 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-xks-ca\") pod \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " Oct 30 21:59:41.714539 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.714285 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-openshift-ca\") pod \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\" (UID: \"dcd74a57-2cf4-4826-ad00-e6657d2c2590\") " Oct 30 21:59:41.714723 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.714700 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-openshift-ca" pod "dcd74a57-2cf4-4826-ad00-e6657d2c2590" (UID: "dcd74a57-2cf4-4826-ad00-e6657d2c2590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:41.716682 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.716653 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-xks-ca" pod "dcd74a57-2cf4-4826-ad00-e6657d2c2590" (UID: "dcd74a57-2cf4-4826-ad00-e6657d2c2590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:41.716778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.716702 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd74a57-2cf4-4826-ad00-e6657d2c2590-kube-api-access-mskfz" pod "dcd74a57-2cf4-4826-ad00-e6657d2c2590" (UID: "dcd74a57-2cf4-4826-ad00-e6657d2c2590"). InnerVolumeSpecName "kube-api-access-mskfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:41.815611 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.815580 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mskfz\" (UniqueName: \"kubernetes.io/projected/dcd74a57-2cf4-4826-ad00-e6657d2c2590-kube-api-access-mskfz\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:41.815611 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.815608 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:41.815611 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:41.815617 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd74a57-2cf4-4826-ad00-e6657d2c2590-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:42.439831 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:42.439798 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" Oct 30 21:59:42.440038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:42.439831 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56" event={"ID":"dcd74a57-2cf4-4826-ad00-e6657d2c2590","Type":"ContainerDied","Data":"798295ff015b87b953a0ce3513a18e38ea93e4b8118062b86422c945a4fae558"} Oct 30 21:59:42.440038 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:42.439881 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798295ff015b87b953a0ce3513a18e38ea93e4b8118062b86422c945a4fae558" Oct 30 21:59:42.477885 ip-10-0-128-26 kubenswrapper[4526]: E1030 21:59:42.477841 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 21:59:49.378154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:49.378123 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh"] Oct 30 21:59:49.490197 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:49.490152 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl"] Oct 30 21:59:49.687598 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:49.687518 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b"] Oct 30 21:59:49.690764 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:49.690739 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1b5b5f-lx75b"] Oct 30 21:59:49.985148 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:49.985057 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b" path="/var/lib/kubelet/pods/c01d6b98-0a0b-446f-8a9a-c7d4e569fb7b/volumes" Oct 30 21:59:52.314838 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:52.314810 4526 scope.go:122] "RemoveContainer" containerID="2b8c5cc9c9792198c213061f311738945e3c2a6cfd596bc8bfa80c3f05810c3c" Oct 30 21:59:52.325352 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:52.325333 4526 scope.go:122] "RemoveContainer" containerID="8e20cb064640947b4d68b98d101b037ca649982506f1e31a4cc7f60eae2c202f" Oct 30 21:59:54.059254 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:54.059217 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn"] Oct 30 21:59:54.108967 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:54.108887 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg"] Oct 30 21:59:54.111873 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:54.111844 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9f1b94-q72kn"] Oct 30 21:59:54.114285 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:54.114261 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b96f5c85-5w2jg"] Oct 30 21:59:54.379135 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:54.379024 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" podUID="51a617e0-ca59-4e58-8e57-b2dbaa6b928b" containerName="payload-processing" containerID="cri-o://323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5" gracePeriod=25 Oct 30 21:59:54.491407 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:54.491362 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" podUID="268a38be-308b-4893-a460-451e17313fac" containerName="payload-pre-processing" containerID="cri-o://785c2684ee8c5c16ae6289dbd2db3bd7f7400045a0db43cd2b6abdadd5c532ba" gracePeriod=25 Oct 30 21:59:55.984709 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:55.984673 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f408a1-4cc4-455d-ae9b-07b577205888" path="/var/lib/kubelet/pods/05f408a1-4cc4-455d-ae9b-07b577205888/volumes" Oct 30 21:59:55.985130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:55.985117 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf32042-919c-4a6d-b379-676feb9106b7" path="/var/lib/kubelet/pods/3bf32042-919c-4a6d-b379-676feb9106b7/volumes" Oct 30 21:59:56.050647 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.050608 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn"] Oct 30 21:59:56.051105 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.051094 4526 cpu_manager.go:414] "containerMap: removing container" podUID="3d1c303c-e1ac-4e5e-bfd0-a66142a14770" containerName="payload-processing" Oct 30 21:59:56.051150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.051108 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1c303c-e1ac-4e5e-bfd0-a66142a14770" containerName="payload-processing" Oct 30 21:59:56.051150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.051123 4526 cpu_manager.go:414] "containerMap: removing container" podUID="dcd74a57-2cf4-4826-ad00-e6657d2c2590" containerName="revoke-keys" Oct 30 21:59:56.051150 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.051128 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd74a57-2cf4-4826-ad00-e6657d2c2590" containerName="revoke-keys" Oct 30 21:59:56.051256 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.051216 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1c303c-e1ac-4e5e-bfd0-a66142a14770" containerName="payload-processing" Oct 30 21:59:56.051256 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.051224 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd74a57-2cf4-4826-ad00-e6657d2c2590" containerName="revoke-keys" Oct 30 21:59:56.056233 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.056208 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.108362 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.108332 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1b5b5f-openshift-default-dockercfg-kzsbh\"" Oct 30 21:59:56.160997 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.160959 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn"] Oct 30 21:59:56.178195 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178162 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f7677-f719-4a14-b94d-7c726309ed0c-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.178411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178209 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.178411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178325 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.178411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178366 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.178411 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178394 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.178746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178421 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.178746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178568 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.178746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178613 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.178746 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.178632 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-kube-api-access-gskkj\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.279998 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280053 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280099 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280085 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-kube-api-access-gskkj\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280131 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f7677-f719-4a14-b94d-7c726309ed0c-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280161 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280212 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280293 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280357 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280409 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280398 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280476 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280559 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280778 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280757 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.280956 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.280802 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.281062 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.281045 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f7677-f719-4a14-b94d-7c726309ed0c-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.282953 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.282930 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.283058 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.282929 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.337589 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.337556 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.337745 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.337597 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-kube-api-access-gskkj\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.369956 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.369923 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:56.805091 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.805064 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn"] Oct 30 21:59:56.807261 ip-10-0-128-26 kubenswrapper[4526]: W1030 21:59:56.807232 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1f7677_f719_4a14_b94d_7c726309ed0c.slice/crio-35250c17fed91b6501c0de8ec1b5d4bb2948c28785637f7074b31e051a3be185 WatchSource:0}: Error finding container 35250c17fed91b6501c0de8ec1b5d4bb2948c28785637f7074b31e051a3be185: Status 404 returned error can't find the container with id 35250c17fed91b6501c0de8ec1b5d4bb2948c28785637f7074b31e051a3be185 Oct 30 21:59:56.809951 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.809921 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:56.810026 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.809987 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:56.810065 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:56.810030 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:57.386252 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:57.386212 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn"] Oct 30 21:59:57.543130 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:57.543094 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" event={"ID":"2e1f7677-f719-4a14-b94d-7c726309ed0c","Type":"ContainerStarted","Data":"c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac"} Oct 30 21:59:57.543323 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:57.543149 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" event={"ID":"2e1f7677-f719-4a14-b94d-7c726309ed0c","Type":"ContainerStarted","Data":"35250c17fed91b6501c0de8ec1b5d4bb2948c28785637f7074b31e051a3be185"} Oct 30 21:59:57.543368 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:57.543326 4526 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" secret="" err="secret \"e2e-worker-w0-1b5b5f-openshift-default-dockercfg-kzsbh\" not found" Oct 30 21:59:57.663032 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:57.662925 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" podStartSLOduration=1.662909285 podStartE2EDuration="1.662909285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:57.660538498 +0000 UTC m=+2587.793911694" watchObservedRunningTime="2026-10-30 21:59:57.662909285 +0000 UTC m=+2587.796282483" Oct 30 21:59:58.370154 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:58.370120 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:58.376015 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:58.375982 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:58.550349 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:58.550296 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" podUID="2e1f7677-f719-4a14-b94d-7c726309ed0c" containerName="istio-proxy" containerID="cri-o://c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac" gracePeriod=30 Oct 30 21:59:58.550905 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:58.550424 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 21:59:58.551641 ip-10-0-128-26 kubenswrapper[4526]: I1030 21:59:58.551613 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 22:00:00.170796 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.170761 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql"] Oct 30 22:00:00.174739 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.174720 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:00.181115 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.181091 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584"] Oct 30 22:00:00.185110 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.185088 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:00.185711 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.185688 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724"] Oct 30 22:00:00.189515 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.189480 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:00.228822 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.228792 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:00:00.278125 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.278071 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql"] Oct 30 22:00:00.281804 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.281774 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584"] Oct 30 22:00:00.291692 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.291659 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724"] Oct 30 22:00:00.327443 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.327412 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e055e9-48cc-4405-ba98-35a85e1295f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584\" (UID: \"54e055e9-48cc-4405-ba98-35a85e1295f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:00.327666 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.327580 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78gtz\" (UniqueName: \"kubernetes.io/projected/df43a187-aaba-40ac-aa42-c60f8d53f263-kube-api-access-78gtz\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql\" (UID: \"df43a187-aaba-40ac-aa42-c60f8d53f263\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:00.327746 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.327688 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724\" (UID: \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:00.327813 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.327798 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqg2\" (UniqueName: \"kubernetes.io/projected/54e055e9-48cc-4405-ba98-35a85e1295f3-kube-api-access-6qqg2\") pod \"maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584\" (UID: \"54e055e9-48cc-4405-ba98-35a85e1295f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:00.327945 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.327926 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df43a187-aaba-40ac-aa42-c60f8d53f263-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql\" (UID: \"df43a187-aaba-40ac-aa42-c60f8d53f263\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:00.327984 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.327962 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5q8\" (UniqueName: \"kubernetes.io/projected/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-kube-api-access-7b5q8\") pod \"maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724\" (UID: \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:00.429611 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.429528 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78gtz\" (UniqueName: \"kubernetes.io/projected/df43a187-aaba-40ac-aa42-c60f8d53f263-kube-api-access-78gtz\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql\" (UID: \"df43a187-aaba-40ac-aa42-c60f8d53f263\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:00.429611 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.429573 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724\" (UID: \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:00.429611 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.429606 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqg2\" (UniqueName: \"kubernetes.io/projected/54e055e9-48cc-4405-ba98-35a85e1295f3-kube-api-access-6qqg2\") pod \"maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584\" (UID: \"54e055e9-48cc-4405-ba98-35a85e1295f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:00.429904 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.429676 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df43a187-aaba-40ac-aa42-c60f8d53f263-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql\" (UID: \"df43a187-aaba-40ac-aa42-c60f8d53f263\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:00.429904 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.429701 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5q8\" (UniqueName: \"kubernetes.io/projected/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-kube-api-access-7b5q8\") pod \"maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724\" (UID: \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:00.429904 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.429844 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e055e9-48cc-4405-ba98-35a85e1295f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584\" (UID: \"54e055e9-48cc-4405-ba98-35a85e1295f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:00.430303 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.430282 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724\" (UID: \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:00.430339 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.430316 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df43a187-aaba-40ac-aa42-c60f8d53f263-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql\" (UID: \"df43a187-aaba-40ac-aa42-c60f8d53f263\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:00.430441 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.430425 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e055e9-48cc-4405-ba98-35a85e1295f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584\" (UID: \"54e055e9-48cc-4405-ba98-35a85e1295f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:00.493739 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.493705 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5q8\" (UniqueName: \"kubernetes.io/projected/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-kube-api-access-7b5q8\") pod \"maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724\" (UID: \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:00.494968 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.494945 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78gtz\" (UniqueName: \"kubernetes.io/projected/df43a187-aaba-40ac-aa42-c60f8d53f263-kube-api-access-78gtz\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql\" (UID: \"df43a187-aaba-40ac-aa42-c60f8d53f263\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:00.497824 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.497795 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqg2\" (UniqueName: \"kubernetes.io/projected/54e055e9-48cc-4405-ba98-35a85e1295f3-kube-api-access-6qqg2\") pod \"maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584\" (UID: \"54e055e9-48cc-4405-ba98-35a85e1295f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:00.498605 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.498587 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:00.505915 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.505895 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:00.788825 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.788799 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:00.885744 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.885702 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584"] Oct 30 22:00:00.886513 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:00:00.886471 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e055e9_48cc_4405_ba98_35a85e1295f3.slice/crio-d1998399e4aa14bd3c17ceac2eb059a5efc398c42566d1cd9d324b978e1a8711 WatchSource:0}: Error finding container d1998399e4aa14bd3c17ceac2eb059a5efc398c42566d1cd9d324b978e1a8711: Status 404 returned error can't find the container with id d1998399e4aa14bd3c17ceac2eb059a5efc398c42566d1cd9d324b978e1a8711 Oct 30 22:00:00.933172 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:00.933058 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724"] Oct 30 22:00:00.934525 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:00:00.934456 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ccae58_f3c9_4810_8c48_cfc3c3511e4d.slice/crio-7eaef0184e71eb56be7056cebc7c3ab5014a8683846391d8f26758c721a2a38b WatchSource:0}: Error finding container 7eaef0184e71eb56be7056cebc7c3ab5014a8683846391d8f26758c721a2a38b: Status 404 returned error can't find the container with id 7eaef0184e71eb56be7056cebc7c3ab5014a8683846391d8f26758c721a2a38b Oct 30 22:00:01.171147 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.171119 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql"] Oct 30 22:00:01.172466 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:00:01.172439 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf43a187_aaba_40ac_aa42_c60f8d53f263.slice/crio-1d78c6e7ed919f00caaf772911f63e6b228cd722fa73e8e2813b91f690839534 WatchSource:0}: Error finding container 1d78c6e7ed919f00caaf772911f63e6b228cd722fa73e8e2813b91f690839534: Status 404 returned error can't find the container with id 1d78c6e7ed919f00caaf772911f63e6b228cd722fa73e8e2813b91f690839534 Oct 30 22:00:01.574825 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.574785 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" event={"ID":"22ccae58-f3c9-4810-8c48-cfc3c3511e4d","Type":"ContainerStarted","Data":"b1a076a295f0a101e4b6e1c0bb8977ba73b696e8f040a8ad88efc6aa8e01373d"} Oct 30 22:00:01.575034 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.574858 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" event={"ID":"22ccae58-f3c9-4810-8c48-cfc3c3511e4d","Type":"ContainerStarted","Data":"7eaef0184e71eb56be7056cebc7c3ab5014a8683846391d8f26758c721a2a38b"} Oct 30 22:00:01.576401 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.576357 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" event={"ID":"df43a187-aaba-40ac-aa42-c60f8d53f263","Type":"ContainerStarted","Data":"07f5a2cb4e98a4442fd6a2b837a21e29c06b5798d19a870817ef050478a2dc0a"} Oct 30 22:00:01.576560 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.576408 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" event={"ID":"df43a187-aaba-40ac-aa42-c60f8d53f263","Type":"ContainerStarted","Data":"1d78c6e7ed919f00caaf772911f63e6b228cd722fa73e8e2813b91f690839534"} Oct 30 22:00:01.578097 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.578075 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" event={"ID":"54e055e9-48cc-4405-ba98-35a85e1295f3","Type":"ContainerStarted","Data":"41645f0d8ab3568becece37b412774b905aee48f4bff2174d64de0f3d179a54a"} Oct 30 22:00:01.578213 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.578116 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" event={"ID":"54e055e9-48cc-4405-ba98-35a85e1295f3","Type":"ContainerStarted","Data":"d1998399e4aa14bd3c17ceac2eb059a5efc398c42566d1cd9d324b978e1a8711"} Oct 30 22:00:01.696002 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.695957 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" podStartSLOduration=1.695941505 podStartE2EDuration="1.695941505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.692892741 +0000 UTC m=+2591.826265940" watchObservedRunningTime="2026-10-30 22:00:01.695941505 +0000 UTC m=+2591.829314727" Oct 30 22:00:01.809588 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.809531 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" podStartSLOduration=1.809516055 podStartE2EDuration="1.809516055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.808923222 +0000 UTC m=+2591.942296414" watchObservedRunningTime="2026-10-30 22:00:01.809516055 +0000 UTC m=+2591.942889250" Oct 30 22:00:01.934006 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:01.933950 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" podStartSLOduration=1.933935132 podStartE2EDuration="1.933935132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.932324479 +0000 UTC m=+2592.065697672" watchObservedRunningTime="2026-10-30 22:00:01.933935132 +0000 UTC m=+2592.067308327" Oct 30 22:00:02.483709 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.483603 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" podUID="51a617e0-ca59-4e58-8e57-b2dbaa6b928b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:00:02.483709 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.483641 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" podUID="268a38be-308b-4893-a460-451e17313fac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:00:02.586652 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.586620 4526 generic.go:356] "Generic (PLEG): container finished" podID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerID="b1a076a295f0a101e4b6e1c0bb8977ba73b696e8f040a8ad88efc6aa8e01373d" exitCode=60 Oct 30 22:00:02.586985 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.586712 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" event={"ID":"22ccae58-f3c9-4810-8c48-cfc3c3511e4d","Type":"ContainerDied","Data":"b1a076a295f0a101e4b6e1c0bb8977ba73b696e8f040a8ad88efc6aa8e01373d"} Oct 30 22:00:02.586985 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.586969 4526 scope.go:122] "RemoveContainer" containerID="b1a076a295f0a101e4b6e1c0bb8977ba73b696e8f040a8ad88efc6aa8e01373d" Oct 30 22:00:02.588888 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.588844 4526 generic.go:356] "Generic (PLEG): container finished" podID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerID="07f5a2cb4e98a4442fd6a2b837a21e29c06b5798d19a870817ef050478a2dc0a" exitCode=60 Oct 30 22:00:02.588989 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.588881 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" event={"ID":"df43a187-aaba-40ac-aa42-c60f8d53f263","Type":"ContainerDied","Data":"07f5a2cb4e98a4442fd6a2b837a21e29c06b5798d19a870817ef050478a2dc0a"} Oct 30 22:00:02.589169 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.589150 4526 scope.go:122] "RemoveContainer" containerID="07f5a2cb4e98a4442fd6a2b837a21e29c06b5798d19a870817ef050478a2dc0a" Oct 30 22:00:02.590817 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.590796 4526 generic.go:356] "Generic (PLEG): container finished" podID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerID="41645f0d8ab3568becece37b412774b905aee48f4bff2174d64de0f3d179a54a" exitCode=60 Oct 30 22:00:02.590919 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.590846 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" event={"ID":"54e055e9-48cc-4405-ba98-35a85e1295f3","Type":"ContainerDied","Data":"41645f0d8ab3568becece37b412774b905aee48f4bff2174d64de0f3d179a54a"} Oct 30 22:00:02.591201 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:02.591093 4526 scope.go:122] "RemoveContainer" containerID="41645f0d8ab3568becece37b412774b905aee48f4bff2174d64de0f3d179a54a" Oct 30 22:00:03.599534 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.599502 4526 generic.go:356] "Generic (PLEG): container finished" podID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerID="bf4a7cb93dea5e355940b5158ea8975199fb416ae293cebc372f8517ffd4741a" exitCode=60 Oct 30 22:00:03.599534 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.599528 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" event={"ID":"22ccae58-f3c9-4810-8c48-cfc3c3511e4d","Type":"ContainerDied","Data":"bf4a7cb93dea5e355940b5158ea8975199fb416ae293cebc372f8517ffd4741a"} Oct 30 22:00:03.600082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.599587 4526 scope.go:122] "RemoveContainer" containerID="b1a076a295f0a101e4b6e1c0bb8977ba73b696e8f040a8ad88efc6aa8e01373d" Oct 30 22:00:03.600082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.599944 4526 scope.go:122] "RemoveContainer" containerID="bf4a7cb93dea5e355940b5158ea8975199fb416ae293cebc372f8517ffd4741a" Oct 30 22:00:03.600314 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:03.600286 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724_odh-ai-gateway-infra(22ccae58-f3c9-4810-8c48-cfc3c3511e4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" Oct 30 22:00:03.601767 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.601745 4526 generic.go:356] "Generic (PLEG): container finished" podID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerID="33c4d7f2513dbb941044d379bc4c220bed28da7f74eb8a634d51f25595133e09" exitCode=60 Oct 30 22:00:03.601890 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.601797 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" event={"ID":"df43a187-aaba-40ac-aa42-c60f8d53f263","Type":"ContainerDied","Data":"33c4d7f2513dbb941044d379bc4c220bed28da7f74eb8a634d51f25595133e09"} Oct 30 22:00:03.602127 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.602109 4526 scope.go:122] "RemoveContainer" containerID="33c4d7f2513dbb941044d379bc4c220bed28da7f74eb8a634d51f25595133e09" Oct 30 22:00:03.602427 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:03.602405 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql_odh-ai-gateway-infra(df43a187-aaba-40ac-aa42-c60f8d53f263)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" Oct 30 22:00:03.604052 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.604029 4526 generic.go:356] "Generic (PLEG): container finished" podID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerID="2d7154289c8c7a8df0c50d85de081fbc3f1392e1680c1494a58a62fcf3f9d53a" exitCode=60 Oct 30 22:00:03.604171 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.604072 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" event={"ID":"54e055e9-48cc-4405-ba98-35a85e1295f3","Type":"ContainerDied","Data":"2d7154289c8c7a8df0c50d85de081fbc3f1392e1680c1494a58a62fcf3f9d53a"} Oct 30 22:00:03.604367 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.604351 4526 scope.go:122] "RemoveContainer" containerID="2d7154289c8c7a8df0c50d85de081fbc3f1392e1680c1494a58a62fcf3f9d53a" Oct 30 22:00:03.604655 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:03.604631 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584_odh-ai-gateway-infra(54e055e9-48cc-4405-ba98-35a85e1295f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" Oct 30 22:00:03.688977 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.688948 4526 scope.go:122] "RemoveContainer" containerID="07f5a2cb4e98a4442fd6a2b837a21e29c06b5798d19a870817ef050478a2dc0a" Oct 30 22:00:03.705443 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.705421 4526 scope.go:122] "RemoveContainer" containerID="41645f0d8ab3568becece37b412774b905aee48f4bff2174d64de0f3d179a54a" Oct 30 22:00:03.829362 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.829332 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 22:00:03.968757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.968650 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-credential-socket\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.968757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.968718 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-socket\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.969014 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.968776 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-token\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-token\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.969014 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.968836 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-envoy\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.969014 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.968982 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-podinfo\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.969014 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.968973 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-credential-socket" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.969014 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969012 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-data\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.969284 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969032 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-socket" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.969284 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969077 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-kube-api-access-gskkj\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.969284 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969127 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-certs\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.969284 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969168 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1f7677-f719-4a14-b94d-7c726309ed0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f7677-f719-4a14-b94d-7c726309ed0c-istiod-ca-cert\") pod \"2e1f7677-f719-4a14-b94d-7c726309ed0c\" (UID: \"2e1f7677-f719-4a14-b94d-7c726309ed0c\") " Oct 30 22:00:03.969547 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969313 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-data" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.969623 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969523 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-certs" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.969701 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969679 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1f7677-f719-4a14-b94d-7c726309ed0c-istiod-ca-cert" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:03.970000 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969950 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.970000 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969979 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.970000 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.969997 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e1f7677-f719-4a14-b94d-7c726309ed0c-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.970160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.970012 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.970160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.970026 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.971536 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.971508 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-token" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:03.971646 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.971537 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-podinfo" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:03.971763 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.971742 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-envoy" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.971903 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:03.971883 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-kube-api-access-gskkj" pod "2e1f7677-f719-4a14-b94d-7c726309ed0c" (UID: "2e1f7677-f719-4a14-b94d-7c726309ed0c"). InnerVolumeSpecName "kube-api-access-gskkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:04.071217 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.071181 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:04.071217 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.071211 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:04.071217 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.071222 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e1f7677-f719-4a14-b94d-7c726309ed0c-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:04.071589 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.071232 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gskkj\" (UniqueName: \"kubernetes.io/projected/2e1f7677-f719-4a14-b94d-7c726309ed0c-kube-api-access-gskkj\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:04.613541 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.613144 4526 scope.go:122] "RemoveContainer" containerID="2d7154289c8c7a8df0c50d85de081fbc3f1392e1680c1494a58a62fcf3f9d53a" Oct 30 22:00:04.613541 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:04.613511 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584_odh-ai-gateway-infra(54e055e9-48cc-4405-ba98-35a85e1295f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" Oct 30 22:00:04.614532 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.614479 4526 generic.go:356] "Generic (PLEG): container finished" podID="2e1f7677-f719-4a14-b94d-7c726309ed0c" containerID="c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac" exitCode=0 Oct 30 22:00:04.614532 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.614517 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" event={"ID":"2e1f7677-f719-4a14-b94d-7c726309ed0c","Type":"ContainerDied","Data":"c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac"} Oct 30 22:00:04.614532 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.614538 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" Oct 30 22:00:04.614811 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.614576 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn" event={"ID":"2e1f7677-f719-4a14-b94d-7c726309ed0c","Type":"ContainerDied","Data":"35250c17fed91b6501c0de8ec1b5d4bb2948c28785637f7074b31e051a3be185"} Oct 30 22:00:04.614811 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.614599 4526 scope.go:122] "RemoveContainer" containerID="c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac" Oct 30 22:00:04.617102 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.617079 4526 scope.go:122] "RemoveContainer" containerID="bf4a7cb93dea5e355940b5158ea8975199fb416ae293cebc372f8517ffd4741a" Oct 30 22:00:04.617635 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:04.617595 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724_odh-ai-gateway-infra(22ccae58-f3c9-4810-8c48-cfc3c3511e4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" Oct 30 22:00:04.618933 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.618913 4526 scope.go:122] "RemoveContainer" containerID="33c4d7f2513dbb941044d379bc4c220bed28da7f74eb8a634d51f25595133e09" Oct 30 22:00:04.619148 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:04.619131 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql_odh-ai-gateway-infra(df43a187-aaba-40ac-aa42-c60f8d53f263)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" Oct 30 22:00:04.625254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.625230 4526 scope.go:122] "RemoveContainer" containerID="c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac" Oct 30 22:00:04.625524 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:04.625505 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac\": container with ID starting with c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac not found: ID does not exist" containerID="c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac" Oct 30 22:00:04.625590 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.625533 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac"} err="failed to get container status \"c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac\": rpc error: code = NotFound desc = could not find container \"c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac\": container with ID starting with c517a63c0a9110544b2a17bcd3102770e921567d34b737dbcf6dded733bb96ac not found: ID does not exist" Oct 30 22:00:04.849302 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.849265 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn"] Oct 30 22:00:04.862258 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:04.862221 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-rhshn"] Oct 30 22:00:05.985878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:05.985843 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1f7677-f719-4a14-b94d-7c726309ed0c" path="/var/lib/kubelet/pods/2e1f7677-f719-4a14-b94d-7c726309ed0c/volumes" Oct 30 22:00:12.477940 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:12.477901 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:00:12.483379 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:12.483347 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" podUID="51a617e0-ca59-4e58-8e57-b2dbaa6b928b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:00:12.483544 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:12.483379 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" podUID="268a38be-308b-4893-a460-451e17313fac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:00:14.651482 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.651413 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 22:00:14.689259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.689225 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-plugins-config-volume\") pod \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\" (UID: \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\") " Oct 30 22:00:14.689796 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.689743 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-plugins-config-volume" pod "51a617e0-ca59-4e58-8e57-b2dbaa6b928b" (UID: "51a617e0-ca59-4e58-8e57-b2dbaa6b928b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:14.709417 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.709375 4526 generic.go:356] "Generic (PLEG): container finished" podID="268a38be-308b-4893-a460-451e17313fac" containerID="785c2684ee8c5c16ae6289dbd2db3bd7f7400045a0db43cd2b6abdadd5c532ba" exitCode=0 Oct 30 22:00:14.709631 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.709556 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" event={"ID":"268a38be-308b-4893-a460-451e17313fac","Type":"ContainerDied","Data":"785c2684ee8c5c16ae6289dbd2db3bd7f7400045a0db43cd2b6abdadd5c532ba"} Oct 30 22:00:14.713641 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.713607 4526 generic.go:356] "Generic (PLEG): container finished" podID="51a617e0-ca59-4e58-8e57-b2dbaa6b928b" containerID="323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5" exitCode=0 Oct 30 22:00:14.713802 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.713686 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" Oct 30 22:00:14.713802 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.713686 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" event={"ID":"51a617e0-ca59-4e58-8e57-b2dbaa6b928b","Type":"ContainerDied","Data":"323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5"} Oct 30 22:00:14.713802 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.713760 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh" event={"ID":"51a617e0-ca59-4e58-8e57-b2dbaa6b928b","Type":"ContainerDied","Data":"49171d3b23e7ac7791ea55ce2dd9dfbe9f21bdc50633c9684fdae3aa4711b0d9"} Oct 30 22:00:14.713802 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.713789 4526 scope.go:122] "RemoveContainer" containerID="323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5" Oct 30 22:00:14.724671 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.724648 4526 scope.go:122] "RemoveContainer" containerID="323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5" Oct 30 22:00:14.725037 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:14.725006 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5\": container with ID starting with 323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5 not found: ID does not exist" containerID="323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5" Oct 30 22:00:14.725140 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.725050 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5"} err="failed to get container status \"323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5\": rpc error: code = NotFound desc = could not find container \"323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5\": container with ID starting with 323a68f8e3518c360ee56941a8451ec360d4bb2b91e7646d820d450fa4691ec5 not found: ID does not exist" Oct 30 22:00:14.765021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.764989 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 22:00:14.791513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.790883 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268a38be-308b-4893-a460-451e17313fac-kube-api-access-t2mtr\" (UniqueName: \"kubernetes.io/projected/268a38be-308b-4893-a460-451e17313fac-kube-api-access-t2mtr\") pod \"268a38be-308b-4893-a460-451e17313fac\" (UID: \"268a38be-308b-4893-a460-451e17313fac\") " Oct 30 22:00:14.791513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.790980 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268a38be-308b-4893-a460-451e17313fac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a38be-308b-4893-a460-451e17313fac-plugins-config-volume\") pod \"268a38be-308b-4893-a460-451e17313fac\" (UID: \"268a38be-308b-4893-a460-451e17313fac\") " Oct 30 22:00:14.791513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.791122 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-kube-api-access-zwhpd\" (UniqueName: \"kubernetes.io/projected/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-kube-api-access-zwhpd\") pod \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\" (UID: \"51a617e0-ca59-4e58-8e57-b2dbaa6b928b\") " Oct 30 22:00:14.791817 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.791546 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:14.792322 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.792291 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268a38be-308b-4893-a460-451e17313fac-plugins-config-volume" pod "268a38be-308b-4893-a460-451e17313fac" (UID: "268a38be-308b-4893-a460-451e17313fac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:14.795902 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.795871 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268a38be-308b-4893-a460-451e17313fac-kube-api-access-t2mtr" pod "268a38be-308b-4893-a460-451e17313fac" (UID: "268a38be-308b-4893-a460-451e17313fac"). InnerVolumeSpecName "kube-api-access-t2mtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:14.796053 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.795925 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-kube-api-access-zwhpd" pod "51a617e0-ca59-4e58-8e57-b2dbaa6b928b" (UID: "51a617e0-ca59-4e58-8e57-b2dbaa6b928b"). InnerVolumeSpecName "kube-api-access-zwhpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:14.893174 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.893128 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2mtr\" (UniqueName: \"kubernetes.io/projected/268a38be-308b-4893-a460-451e17313fac-kube-api-access-t2mtr\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:14.893174 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.893166 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268a38be-308b-4893-a460-451e17313fac-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:14.893174 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.893181 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwhpd\" (UniqueName: \"kubernetes.io/projected/51a617e0-ca59-4e58-8e57-b2dbaa6b928b-kube-api-access-zwhpd\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:14.979052 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.978959 4526 scope.go:122] "RemoveContainer" containerID="2d7154289c8c7a8df0c50d85de081fbc3f1392e1680c1494a58a62fcf3f9d53a" Oct 30 22:00:14.979240 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:14.979163 4526 scope.go:122] "RemoveContainer" containerID="bf4a7cb93dea5e355940b5158ea8975199fb416ae293cebc372f8517ffd4741a" Oct 30 22:00:15.147428 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.147390 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh"] Oct 30 22:00:15.153592 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.153555 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-wwskh"] Oct 30 22:00:15.726629 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.726541 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" event={"ID":"22ccae58-f3c9-4810-8c48-cfc3c3511e4d","Type":"ContainerStarted","Data":"b8fede3053f6b4820f2b00019f58d9254dbb4f2dbf1a28083fbc613af7fd33be"} Oct 30 22:00:15.727971 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.727952 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" Oct 30 22:00:15.728125 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.727992 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl" event={"ID":"268a38be-308b-4893-a460-451e17313fac","Type":"ContainerDied","Data":"3d0b4903846a2640a68821c3a2c1aed0eeccaccad0c59520610dadcdd9c05814"} Oct 30 22:00:15.728125 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.728052 4526 scope.go:122] "RemoveContainer" containerID="785c2684ee8c5c16ae6289dbd2db3bd7f7400045a0db43cd2b6abdadd5c532ba" Oct 30 22:00:15.730517 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.730473 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" event={"ID":"54e055e9-48cc-4405-ba98-35a85e1295f3","Type":"ContainerStarted","Data":"122075274be36690f4fb31a90bd8dde82fe4b6ad458fa6a0ebf854a8f814322b"} Oct 30 22:00:15.985103 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.984997 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a617e0-ca59-4e58-8e57-b2dbaa6b928b" path="/var/lib/kubelet/pods/51a617e0-ca59-4e58-8e57-b2dbaa6b928b/volumes" Oct 30 22:00:15.985925 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:15.985907 4526 scope.go:122] "RemoveContainer" containerID="33c4d7f2513dbb941044d379bc4c220bed28da7f74eb8a634d51f25595133e09" Oct 30 22:00:16.088734 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.088696 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl"] Oct 30 22:00:16.091246 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.091222 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-hzwkl"] Oct 30 22:00:16.739173 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.739081 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" event={"ID":"df43a187-aaba-40ac-aa42-c60f8d53f263","Type":"ContainerStarted","Data":"0f5c0d78f8609eb2b1884b0f2d70c68628cff406d8cc2d2172e592e1b15cbbef"} Oct 30 22:00:16.740937 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.740916 4526 generic.go:356] "Generic (PLEG): container finished" podID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerID="122075274be36690f4fb31a90bd8dde82fe4b6ad458fa6a0ebf854a8f814322b" exitCode=60 Oct 30 22:00:16.741058 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.740983 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" event={"ID":"54e055e9-48cc-4405-ba98-35a85e1295f3","Type":"ContainerDied","Data":"122075274be36690f4fb31a90bd8dde82fe4b6ad458fa6a0ebf854a8f814322b"} Oct 30 22:00:16.741058 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.741032 4526 scope.go:122] "RemoveContainer" containerID="2d7154289c8c7a8df0c50d85de081fbc3f1392e1680c1494a58a62fcf3f9d53a" Oct 30 22:00:16.741377 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.741361 4526 scope.go:122] "RemoveContainer" containerID="122075274be36690f4fb31a90bd8dde82fe4b6ad458fa6a0ebf854a8f814322b" Oct 30 22:00:16.741708 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:16.741683 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584_odh-ai-gateway-infra(54e055e9-48cc-4405-ba98-35a85e1295f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" Oct 30 22:00:16.743000 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.742979 4526 generic.go:356] "Generic (PLEG): container finished" podID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerID="b8fede3053f6b4820f2b00019f58d9254dbb4f2dbf1a28083fbc613af7fd33be" exitCode=60 Oct 30 22:00:16.743110 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.743005 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" event={"ID":"22ccae58-f3c9-4810-8c48-cfc3c3511e4d","Type":"ContainerDied","Data":"b8fede3053f6b4820f2b00019f58d9254dbb4f2dbf1a28083fbc613af7fd33be"} Oct 30 22:00:16.743281 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.743268 4526 scope.go:122] "RemoveContainer" containerID="b8fede3053f6b4820f2b00019f58d9254dbb4f2dbf1a28083fbc613af7fd33be" Oct 30 22:00:16.743480 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:16.743465 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724_odh-ai-gateway-infra(22ccae58-f3c9-4810-8c48-cfc3c3511e4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" Oct 30 22:00:16.753854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.753837 4526 scope.go:122] "RemoveContainer" containerID="bf4a7cb93dea5e355940b5158ea8975199fb416ae293cebc372f8517ffd4741a" Oct 30 22:00:16.882761 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:16.882720 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724"] Oct 30 22:00:17.025092 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.025000 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584"] Oct 30 22:00:17.756127 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.756095 4526 generic.go:356] "Generic (PLEG): container finished" podID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerID="0f5c0d78f8609eb2b1884b0f2d70c68628cff406d8cc2d2172e592e1b15cbbef" exitCode=60 Oct 30 22:00:17.756649 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.756132 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" event={"ID":"df43a187-aaba-40ac-aa42-c60f8d53f263","Type":"ContainerDied","Data":"0f5c0d78f8609eb2b1884b0f2d70c68628cff406d8cc2d2172e592e1b15cbbef"} Oct 30 22:00:17.756649 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.756234 4526 scope.go:122] "RemoveContainer" containerID="33c4d7f2513dbb941044d379bc4c220bed28da7f74eb8a634d51f25595133e09" Oct 30 22:00:17.756808 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.756702 4526 scope.go:122] "RemoveContainer" containerID="0f5c0d78f8609eb2b1884b0f2d70c68628cff406d8cc2d2172e592e1b15cbbef" Oct 30 22:00:17.757078 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:17.757044 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql_odh-ai-gateway-infra(df43a187-aaba-40ac-aa42-c60f8d53f263)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" Oct 30 22:00:17.897221 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.895364 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql"] Oct 30 22:00:17.955981 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.955942 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:17.958882 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.958863 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:17.985799 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:17.985770 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268a38be-308b-4893-a460-451e17313fac" path="/var/lib/kubelet/pods/268a38be-308b-4893-a460-451e17313fac/volumes" Oct 30 22:00:18.028603 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.028512 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-maas-api-service-ca\") pod \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\" (UID: \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\") " Oct 30 22:00:18.028603 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.028571 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-kube-api-access-7b5q8\" (UniqueName: \"kubernetes.io/projected/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-kube-api-access-7b5q8\") pod \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\" (UID: \"22ccae58-f3c9-4810-8c48-cfc3c3511e4d\") " Oct 30 22:00:18.028869 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.028658 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e055e9-48cc-4405-ba98-35a85e1295f3-kube-api-access-6qqg2\" (UniqueName: \"kubernetes.io/projected/54e055e9-48cc-4405-ba98-35a85e1295f3-kube-api-access-6qqg2\") pod \"54e055e9-48cc-4405-ba98-35a85e1295f3\" (UID: \"54e055e9-48cc-4405-ba98-35a85e1295f3\") " Oct 30 22:00:18.028869 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.028794 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e055e9-48cc-4405-ba98-35a85e1295f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e055e9-48cc-4405-ba98-35a85e1295f3-maas-api-service-ca\") pod \"54e055e9-48cc-4405-ba98-35a85e1295f3\" (UID: \"54e055e9-48cc-4405-ba98-35a85e1295f3\") " Oct 30 22:00:18.028999 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.028949 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-maas-api-service-ca" pod "22ccae58-f3c9-4810-8c48-cfc3c3511e4d" (UID: "22ccae58-f3c9-4810-8c48-cfc3c3511e4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.029228 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.029201 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e055e9-48cc-4405-ba98-35a85e1295f3-maas-api-service-ca" pod "54e055e9-48cc-4405-ba98-35a85e1295f3" (UID: "54e055e9-48cc-4405-ba98-35a85e1295f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.029323 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.029252 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.031598 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.031564 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-kube-api-access-7b5q8" pod "22ccae58-f3c9-4810-8c48-cfc3c3511e4d" (UID: "22ccae58-f3c9-4810-8c48-cfc3c3511e4d"). InnerVolumeSpecName "kube-api-access-7b5q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.031707 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.031603 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e055e9-48cc-4405-ba98-35a85e1295f3-kube-api-access-6qqg2" pod "54e055e9-48cc-4405-ba98-35a85e1295f3" (UID: "54e055e9-48cc-4405-ba98-35a85e1295f3"). InnerVolumeSpecName "kube-api-access-6qqg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.130854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.130818 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e055e9-48cc-4405-ba98-35a85e1295f3-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.130854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.130846 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5q8\" (UniqueName: \"kubernetes.io/projected/22ccae58-f3c9-4810-8c48-cfc3c3511e4d-kube-api-access-7b5q8\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.130854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.130861 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qqg2\" (UniqueName: \"kubernetes.io/projected/54e055e9-48cc-4405-ba98-35a85e1295f3-kube-api-access-6qqg2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.769658 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.769625 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" Oct 30 22:00:18.770022 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.769672 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584" event={"ID":"54e055e9-48cc-4405-ba98-35a85e1295f3","Type":"ContainerDied","Data":"d1998399e4aa14bd3c17ceac2eb059a5efc398c42566d1cd9d324b978e1a8711"} Oct 30 22:00:18.770022 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.769749 4526 scope.go:122] "RemoveContainer" containerID="122075274be36690f4fb31a90bd8dde82fe4b6ad458fa6a0ebf854a8f814322b" Oct 30 22:00:18.771328 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.771304 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" event={"ID":"22ccae58-f3c9-4810-8c48-cfc3c3511e4d","Type":"ContainerDied","Data":"7eaef0184e71eb56be7056cebc7c3ab5014a8683846391d8f26758c721a2a38b"} Oct 30 22:00:18.771647 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.771356 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724" Oct 30 22:00:18.779982 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.779963 4526 scope.go:122] "RemoveContainer" containerID="b8fede3053f6b4820f2b00019f58d9254dbb4f2dbf1a28083fbc613af7fd33be" Oct 30 22:00:18.891185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.891159 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:18.940197 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.940161 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df43a187-aaba-40ac-aa42-c60f8d53f263-kube-api-access-78gtz\" (UniqueName: \"kubernetes.io/projected/df43a187-aaba-40ac-aa42-c60f8d53f263-kube-api-access-78gtz\") pod \"df43a187-aaba-40ac-aa42-c60f8d53f263\" (UID: \"df43a187-aaba-40ac-aa42-c60f8d53f263\") " Oct 30 22:00:18.940367 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.940215 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df43a187-aaba-40ac-aa42-c60f8d53f263-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df43a187-aaba-40ac-aa42-c60f8d53f263-maas-api-service-ca\") pod \"df43a187-aaba-40ac-aa42-c60f8d53f263\" (UID: \"df43a187-aaba-40ac-aa42-c60f8d53f263\") " Oct 30 22:00:18.940572 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.940547 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df43a187-aaba-40ac-aa42-c60f8d53f263-maas-api-service-ca" pod "df43a187-aaba-40ac-aa42-c60f8d53f263" (UID: "df43a187-aaba-40ac-aa42-c60f8d53f263"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.940712 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.940695 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df43a187-aaba-40ac-aa42-c60f8d53f263-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.942571 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:18.942547 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df43a187-aaba-40ac-aa42-c60f8d53f263-kube-api-access-78gtz" pod "df43a187-aaba-40ac-aa42-c60f8d53f263" (UID: "df43a187-aaba-40ac-aa42-c60f8d53f263"). InnerVolumeSpecName "kube-api-access-78gtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.002413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.002377 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584"] Oct 30 22:00:19.007685 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.007656 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e9aa8e-29889960-pq584"] Oct 30 22:00:19.041806 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.041773 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78gtz\" (UniqueName: \"kubernetes.io/projected/df43a187-aaba-40ac-aa42-c60f8d53f263-kube-api-access-78gtz\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.127268 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.127232 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724"] Oct 30 22:00:19.129970 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.129940 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-75c2acf6-29889960-tp724"] Oct 30 22:00:19.781918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.781890 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" Oct 30 22:00:19.781918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.781898 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql" event={"ID":"df43a187-aaba-40ac-aa42-c60f8d53f263","Type":"ContainerDied","Data":"1d78c6e7ed919f00caaf772911f63e6b228cd722fa73e8e2813b91f690839534"} Oct 30 22:00:19.782504 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.781963 4526 scope.go:122] "RemoveContainer" containerID="0f5c0d78f8609eb2b1884b0f2d70c68628cff406d8cc2d2172e592e1b15cbbef" Oct 30 22:00:19.948306 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.948266 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql"] Oct 30 22:00:19.984115 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.984082 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" path="/var/lib/kubelet/pods/22ccae58-f3c9-4810-8c48-cfc3c3511e4d/volumes" Oct 30 22:00:19.984536 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.984523 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" path="/var/lib/kubelet/pods/54e055e9-48cc-4405-ba98-35a85e1295f3/volumes" Oct 30 22:00:19.996122 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:19.996102 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889960-vzkql"] Oct 30 22:00:21.985184 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:21.985150 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" path="/var/lib/kubelet/pods/df43a187-aaba-40ac-aa42-c60f8d53f263/volumes" Oct 30 22:00:36.494560 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:36.494525 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56"] Oct 30 22:00:36.542276 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:36.542247 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-gcv56"] Oct 30 22:00:37.984167 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:37.984134 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd74a57-2cf4-4826-ad00-e6657d2c2590" path="/var/lib/kubelet/pods/dcd74a57-2cf4-4826-ad00-e6657d2c2590/volumes" Oct 30 22:00:42.478017 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:00:42.477975 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:00:52.364979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:52.364944 4526 scope.go:122] "RemoveContainer" containerID="b099f4a414908e1ab0589b0a322880914453857e88591857ac09f3b71186bc99" Oct 30 22:00:52.374345 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:52.374323 4526 scope.go:122] "RemoveContainer" containerID="2494766b655a54889d38313dc1aeb7cc56ff24f62a77a6e255154d47f9068f4d" Oct 30 22:00:52.382434 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:52.382416 4526 scope.go:122] "RemoveContainer" containerID="a9c578db721dee8796b91b61deb5ee43c632b004fa01359d19489b4cc4b8f998" Oct 30 22:00:52.390798 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:52.390774 4526 scope.go:122] "RemoveContainer" containerID="175e72b3ce3ab77ae8599d900a1f29d764dbab2c47356ea04173ae958d8efb95" Oct 30 22:00:52.399109 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:52.399091 4526 scope.go:122] "RemoveContainer" containerID="de7aead38ec1292cfcee807881b5313c79055d10632846d47292bf104a66532c" Oct 30 22:00:52.407376 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:52.407355 4526 scope.go:122] "RemoveContainer" containerID="86905f6d50fabf32da05acb7fd45e1b388c2c10c0d4ee18e934c3a52293708ea" Oct 30 22:00:57.117974 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:57.117928 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv"] Oct 30 22:00:57.167058 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:57.167020 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa324b1-2pdvv"] Oct 30 22:00:57.984819 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:57.984786 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68150331-c4b4-4c27-9a89-be2a545153c2" path="/var/lib/kubelet/pods/68150331-c4b4-4c27-9a89-be2a545153c2/volumes" Oct 30 22:00:59.050061 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:59.050027 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m"] Oct 30 22:00:59.102230 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:59.102184 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fc0d783-k6z8m"] Oct 30 22:00:59.985403 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:00:59.985360 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a2fe3e-8217-4535-9e8f-596ff75e2352" path="/var/lib/kubelet/pods/12a2fe3e-8217-4535-9e8f-596ff75e2352/volumes" Oct 30 22:01:12.480024 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:01:12.479937 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:01:42.477321 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:01:42.477274 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:01:48.939401 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939364 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp"] Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939837 4526 cpu_manager.go:414] "containerMap: removing container" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939848 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939861 4526 cpu_manager.go:414] "containerMap: removing container" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939867 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939874 4526 cpu_manager.go:414] "containerMap: removing container" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939880 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939887 4526 cpu_manager.go:414] "containerMap: removing container" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939892 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939903 4526 cpu_manager.go:414] "containerMap: removing container" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939907 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939918 4526 cpu_manager.go:414] "containerMap: removing container" podUID="268a38be-308b-4893-a460-451e17313fac" containerName="payload-pre-processing" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939924 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268a38be-308b-4893-a460-451e17313fac" containerName="payload-pre-processing" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939934 4526 cpu_manager.go:414] "containerMap: removing container" podUID="2e1f7677-f719-4a14-b94d-7c726309ed0c" containerName="istio-proxy" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939939 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1f7677-f719-4a14-b94d-7c726309ed0c" containerName="istio-proxy" Oct 30 22:01:48.939949 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939957 4526 cpu_manager.go:414] "containerMap: removing container" podUID="51a617e0-ca59-4e58-8e57-b2dbaa6b928b" containerName="payload-processing" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939962 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a617e0-ca59-4e58-8e57-b2dbaa6b928b" containerName="payload-processing" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939973 4526 cpu_manager.go:414] "containerMap: removing container" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.939977 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940035 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940042 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940050 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940056 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a617e0-ca59-4e58-8e57-b2dbaa6b928b" containerName="payload-processing" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940064 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940070 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940076 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940080 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940087 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="268a38be-308b-4893-a460-451e17313fac" containerName="payload-pre-processing" Oct 30 22:01:48.940440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.940093 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1f7677-f719-4a14-b94d-7c726309ed0c" containerName="istio-proxy" Oct 30 22:01:48.943252 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:48.943236 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.002874 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.002840 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl2t4\" (UniqueName: \"kubernetes.io/projected/73ca2f99-7eb7-4bba-998d-1907101ea3b4-kube-api-access-tl2t4\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.003101 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.002905 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.003101 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.003061 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.046482 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.046447 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:01:49.046482 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.046459 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:01:49.049263 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.049238 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp"] Oct 30 22:01:49.104608 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.104571 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl2t4\" (UniqueName: \"kubernetes.io/projected/73ca2f99-7eb7-4bba-998d-1907101ea3b4-kube-api-access-tl2t4\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.105213 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.104658 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.105213 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.104727 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.105582 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.105560 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.107400 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.107380 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.164285 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.164255 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl2t4\" (UniqueName: \"kubernetes.io/projected/73ca2f99-7eb7-4bba-998d-1907101ea3b4-kube-api-access-tl2t4\") pod \"maas-api-revoke-sub-369374b7-tdkgp\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.256630 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.256535 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:49.654784 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:49.654760 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp"] Oct 30 22:01:49.657166 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:01:49.657130 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ca2f99_7eb7_4bba_998d_1907101ea3b4.slice/crio-0245ee996aea184a03541ff9c28b0c897253bb763633d4db978f65c392e23c4d WatchSource:0}: Error finding container 0245ee996aea184a03541ff9c28b0c897253bb763633d4db978f65c392e23c4d: Status 404 returned error can't find the container with id 0245ee996aea184a03541ff9c28b0c897253bb763633d4db978f65c392e23c4d Oct 30 22:01:50.379476 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:50.379436 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" event={"ID":"73ca2f99-7eb7-4bba-998d-1907101ea3b4","Type":"ContainerStarted","Data":"4458bbdfab40a90c213c66213512b178b9fc7e513cf63ce633a021aa0ed55767"} Oct 30 22:01:50.379985 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:50.379570 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" event={"ID":"73ca2f99-7eb7-4bba-998d-1907101ea3b4","Type":"ContainerStarted","Data":"0245ee996aea184a03541ff9c28b0c897253bb763633d4db978f65c392e23c4d"} Oct 30 22:01:50.494710 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:50.494649 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" podStartSLOduration=2.4946324 podStartE2EDuration="2.4946324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:50.493641624 +0000 UTC m=+2700.627014823" watchObservedRunningTime="2026-10-30 22:01:50.4946324 +0000 UTC m=+2700.628005599" Oct 30 22:01:51.387265 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:51.387230 4526 generic.go:356] "Generic (PLEG): container finished" podID="73ca2f99-7eb7-4bba-998d-1907101ea3b4" containerID="4458bbdfab40a90c213c66213512b178b9fc7e513cf63ce633a021aa0ed55767" exitCode=0 Oct 30 22:01:51.387715 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:51.387309 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" event={"ID":"73ca2f99-7eb7-4bba-998d-1907101ea3b4","Type":"ContainerDied","Data":"4458bbdfab40a90c213c66213512b178b9fc7e513cf63ce633a021aa0ed55767"} Oct 30 22:01:52.482091 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.482004 4526 scope.go:122] "RemoveContainer" containerID="62fc7951f857da1f7ae03fa9eddefc645dc5fd677bd7a4620514231f160ed28a" Oct 30 22:01:52.492983 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.492957 4526 scope.go:122] "RemoveContainer" containerID="39487ea0faf40917202fc2ac4f5518e96b35492bdf937be8e761eaef94d87b79" Oct 30 22:01:52.546421 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.546393 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:52.644575 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.644456 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-xks-ca\") pod \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " Oct 30 22:01:52.644575 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.644556 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-openshift-ca\") pod \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " Oct 30 22:01:52.644851 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.644654 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ca2f99-7eb7-4bba-998d-1907101ea3b4-kube-api-access-tl2t4\" (UniqueName: \"kubernetes.io/projected/73ca2f99-7eb7-4bba-998d-1907101ea3b4-kube-api-access-tl2t4\") pod \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\" (UID: \"73ca2f99-7eb7-4bba-998d-1907101ea3b4\") " Oct 30 22:01:52.644981 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.644956 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-openshift-ca" pod "73ca2f99-7eb7-4bba-998d-1907101ea3b4" (UID: "73ca2f99-7eb7-4bba-998d-1907101ea3b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:52.646863 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.646840 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-xks-ca" pod "73ca2f99-7eb7-4bba-998d-1907101ea3b4" (UID: "73ca2f99-7eb7-4bba-998d-1907101ea3b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:52.646972 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.646840 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ca2f99-7eb7-4bba-998d-1907101ea3b4-kube-api-access-tl2t4" pod "73ca2f99-7eb7-4bba-998d-1907101ea3b4" (UID: "73ca2f99-7eb7-4bba-998d-1907101ea3b4"). InnerVolumeSpecName "kube-api-access-tl2t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:52.746372 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.746326 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:01:52.746372 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.746358 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ca2f99-7eb7-4bba-998d-1907101ea3b4-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:01:52.746372 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:52.746369 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl2t4\" (UniqueName: \"kubernetes.io/projected/73ca2f99-7eb7-4bba-998d-1907101ea3b4-kube-api-access-tl2t4\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:01:53.403137 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:53.403109 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" Oct 30 22:01:53.403340 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:53.403144 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp" event={"ID":"73ca2f99-7eb7-4bba-998d-1907101ea3b4","Type":"ContainerDied","Data":"0245ee996aea184a03541ff9c28b0c897253bb763633d4db978f65c392e23c4d"} Oct 30 22:01:53.403340 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:53.403231 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0245ee996aea184a03541ff9c28b0c897253bb763633d4db978f65c392e23c4d" Oct 30 22:01:56.049679 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:56.049648 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht"] Oct 30 22:01:56.099440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:56.099401 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2"] Oct 30 22:01:56.101762 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:56.101738 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f365ec62-rfhht"] Oct 30 22:01:56.103709 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:56.103688 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dad53d0-x5hg2"] Oct 30 22:01:57.984424 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:57.984390 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e377970-46c6-4a2c-ac24-eb4f14116b4a" path="/var/lib/kubelet/pods/7e377970-46c6-4a2c-ac24-eb4f14116b4a/volumes" Oct 30 22:01:57.984963 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:01:57.984942 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a833c9ee-cf5b-4a50-bece-7d4e9be9d096" path="/var/lib/kubelet/pods/a833c9ee-cf5b-4a50-bece-7d4e9be9d096/volumes" Oct 30 22:02:03.587900 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.587854 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89"] Oct 30 22:02:03.588357 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588342 4526 cpu_manager.go:414] "containerMap: removing container" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:02:03.588413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588359 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:02:03.588413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588368 4526 cpu_manager.go:414] "containerMap: removing container" podUID="73ca2f99-7eb7-4bba-998d-1907101ea3b4" containerName="revoke-keys" Oct 30 22:02:03.588413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588373 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ca2f99-7eb7-4bba-998d-1907101ea3b4" containerName="revoke-keys" Oct 30 22:02:03.588413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588390 4526 cpu_manager.go:414] "containerMap: removing container" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:02:03.588413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588395 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df43a187-aaba-40ac-aa42-c60f8d53f263" containerName="cleanup" Oct 30 22:02:03.588413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588406 4526 cpu_manager.go:414] "containerMap: removing container" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:02:03.588413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588411 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:02:03.588766 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588513 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e055e9-48cc-4405-ba98-35a85e1295f3" containerName="cleanup" Oct 30 22:02:03.588766 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588525 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="22ccae58-f3c9-4810-8c48-cfc3c3511e4d" containerName="cleanup" Oct 30 22:02:03.588766 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.588536 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ca2f99-7eb7-4bba-998d-1907101ea3b4" containerName="revoke-keys" Oct 30 22:02:03.593212 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.593196 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.695079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.695046 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:03.695255 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.695081 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:02:03.697740 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.697715 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89"] Oct 30 22:02:03.766103 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.766071 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7rvm\" (UniqueName: \"kubernetes.io/projected/df39b071-bb0c-4c45-9dd7-4d86a2501817-kube-api-access-z7rvm\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.766337 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.766148 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.766337 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.766256 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.867908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.867814 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7rvm\" (UniqueName: \"kubernetes.io/projected/df39b071-bb0c-4c45-9dd7-4d86a2501817-kube-api-access-z7rvm\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.867908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.867871 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.867908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.867905 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.868574 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.868548 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.870542 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.870519 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:03.924954 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:03.924915 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7rvm\" (UniqueName: \"kubernetes.io/projected/df39b071-bb0c-4c45-9dd7-4d86a2501817-kube-api-access-z7rvm\") pod \"maas-api-revoke-sub-960b5194-7gs89\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:04.206052 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:04.205966 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:04.585284 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:04.585252 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89"] Oct 30 22:02:04.585990 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:02:04.585962 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf39b071_bb0c_4c45_9dd7_4d86a2501817.slice/crio-830b85eef4ec5d6eed775db5e0f6e8c7ede1b108f862393325078992e019fc60 WatchSource:0}: Error finding container 830b85eef4ec5d6eed775db5e0f6e8c7ede1b108f862393325078992e019fc60: Status 404 returned error can't find the container with id 830b85eef4ec5d6eed775db5e0f6e8c7ede1b108f862393325078992e019fc60 Oct 30 22:02:05.479770 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:05.479729 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" event={"ID":"df39b071-bb0c-4c45-9dd7-4d86a2501817","Type":"ContainerStarted","Data":"9981fc1eb9e49b53964304fc6968d8d85ce350fb9835e665cbf9686bcaa311bf"} Oct 30 22:02:05.480215 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:05.479801 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" event={"ID":"df39b071-bb0c-4c45-9dd7-4d86a2501817","Type":"ContainerStarted","Data":"830b85eef4ec5d6eed775db5e0f6e8c7ede1b108f862393325078992e019fc60"} Oct 30 22:02:05.593157 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:05.593108 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" podStartSLOduration=2.593089418 podStartE2EDuration="2.593089418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:05.592148133 +0000 UTC m=+2715.725521332" watchObservedRunningTime="2026-10-30 22:02:05.593089418 +0000 UTC m=+2715.726462616" Oct 30 22:02:06.487833 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:06.487795 4526 generic.go:356] "Generic (PLEG): container finished" podID="df39b071-bb0c-4c45-9dd7-4d86a2501817" containerID="9981fc1eb9e49b53964304fc6968d8d85ce350fb9835e665cbf9686bcaa311bf" exitCode=0 Oct 30 22:02:06.488325 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:06.487847 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" event={"ID":"df39b071-bb0c-4c45-9dd7-4d86a2501817","Type":"ContainerDied","Data":"9981fc1eb9e49b53964304fc6968d8d85ce350fb9835e665cbf9686bcaa311bf"} Oct 30 22:02:07.641289 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.641261 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:07.812667 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.812635 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-openshift-ca\") pod \"df39b071-bb0c-4c45-9dd7-4d86a2501817\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " Oct 30 22:02:07.812863 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.812680 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-xks-ca\") pod \"df39b071-bb0c-4c45-9dd7-4d86a2501817\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " Oct 30 22:02:07.812863 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.812714 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df39b071-bb0c-4c45-9dd7-4d86a2501817-kube-api-access-z7rvm\" (UniqueName: \"kubernetes.io/projected/df39b071-bb0c-4c45-9dd7-4d86a2501817-kube-api-access-z7rvm\") pod \"df39b071-bb0c-4c45-9dd7-4d86a2501817\" (UID: \"df39b071-bb0c-4c45-9dd7-4d86a2501817\") " Oct 30 22:02:07.813122 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.813084 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-openshift-ca" pod "df39b071-bb0c-4c45-9dd7-4d86a2501817" (UID: "df39b071-bb0c-4c45-9dd7-4d86a2501817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:07.815118 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.815091 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-xks-ca" pod "df39b071-bb0c-4c45-9dd7-4d86a2501817" (UID: "df39b071-bb0c-4c45-9dd7-4d86a2501817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:07.815226 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.815194 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df39b071-bb0c-4c45-9dd7-4d86a2501817-kube-api-access-z7rvm" pod "df39b071-bb0c-4c45-9dd7-4d86a2501817" (UID: "df39b071-bb0c-4c45-9dd7-4d86a2501817"). InnerVolumeSpecName "kube-api-access-z7rvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:07.914074 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.914035 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:07.914074 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.914066 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df39b071-bb0c-4c45-9dd7-4d86a2501817-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:07.914074 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:07.914077 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7rvm\" (UniqueName: \"kubernetes.io/projected/df39b071-bb0c-4c45-9dd7-4d86a2501817-kube-api-access-z7rvm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:08.502045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:08.502011 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" Oct 30 22:02:08.502253 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:08.502049 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89" event={"ID":"df39b071-bb0c-4c45-9dd7-4d86a2501817","Type":"ContainerDied","Data":"830b85eef4ec5d6eed775db5e0f6e8c7ede1b108f862393325078992e019fc60"} Oct 30 22:02:08.502253 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:08.502120 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="830b85eef4ec5d6eed775db5e0f6e8c7ede1b108f862393325078992e019fc60" Oct 30 22:02:09.062267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.062224 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z"] Oct 30 22:02:09.108744 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.108714 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp"] Oct 30 22:02:09.111892 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.111869 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0efe0b-bpp9z"] Oct 30 22:02:09.113928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.113908 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08cd6e2-8vplp"] Oct 30 22:02:09.382626 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.382542 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h"] Oct 30 22:02:09.383138 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.383117 4526 cpu_manager.go:414] "containerMap: removing container" podUID="df39b071-bb0c-4c45-9dd7-4d86a2501817" containerName="revoke-keys" Oct 30 22:02:09.383253 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.383140 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df39b071-bb0c-4c45-9dd7-4d86a2501817" containerName="revoke-keys" Oct 30 22:02:09.383321 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.383267 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="df39b071-bb0c-4c45-9dd7-4d86a2501817" containerName="revoke-keys" Oct 30 22:02:09.388410 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.388390 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.419619 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.419597 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt"] Oct 30 22:02:09.423191 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.423171 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.469113 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.469089 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h"] Oct 30 22:02:09.497840 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.497805 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt"] Oct 30 22:02:09.532271 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532241 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.532426 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532360 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.532426 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532418 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd96f278-3477-457e-9a09-56fff149ba61-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.532596 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532540 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hn76\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-kube-api-access-6hn76\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.532713 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532696 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.532769 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532729 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshwd\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-kube-api-access-lshwd\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.532820 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532769 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.532895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532823 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.532895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532888 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.532978 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532917 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9935d58c-f632-459c-b66a-1f3843ac59b3-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.532978 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532940 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.532978 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.532958 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.533107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.533004 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.533107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.533023 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.533107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.533098 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd96f278-3477-457e-9a09-56fff149ba61-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.533243 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.533120 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.533243 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.533137 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.533243 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.533198 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.634351 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634274 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.634351 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634316 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd96f278-3477-457e-9a09-56fff149ba61-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.634351 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634334 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hn76\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-kube-api-access-6hn76\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.634704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634363 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.634704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634400 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshwd\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-kube-api-access-lshwd\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.634704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634434 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.634704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634469 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.634704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634554 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.634979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634778 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9935d58c-f632-459c-b66a-1f3843ac59b3-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.634979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634825 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.634979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634870 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.634979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634927 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.634979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634937 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.634979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.634963 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.635313 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635011 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.635313 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635029 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.635313 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635040 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.635313 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635271 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd96f278-3477-457e-9a09-56fff149ba61-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.635313 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635313 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.635618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635316 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.635618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635323 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.635618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635341 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.635618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635382 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.635618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635433 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.635618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635557 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9935d58c-f632-459c-b66a-1f3843ac59b3-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.635943 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635771 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.635943 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.635810 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.636200 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.636176 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd96f278-3477-457e-9a09-56fff149ba61-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.637323 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.637296 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.637323 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.637320 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.637772 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.637753 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd96f278-3477-457e-9a09-56fff149ba61-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.637945 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.637930 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.691459 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.691436 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hn76\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-kube-api-access-6hn76\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.691459 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.691446 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.691964 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.691949 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.692041 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.692021 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshwd\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-kube-api-access-lshwd\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.702976 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.702957 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:09.735616 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.735584 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:09.985744 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.985665 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81e71d9-8c5d-4813-a741-bc2bb1b23a60" path="/var/lib/kubelet/pods/a81e71d9-8c5d-4813-a741-bc2bb1b23a60/volumes" Oct 30 22:02:09.986106 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:09.986093 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3223b35-b76b-4a50-8334-876682df6579" path="/var/lib/kubelet/pods/c3223b35-b76b-4a50-8334-876682df6579/volumes" Oct 30 22:02:10.131124 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.131089 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h"] Oct 30 22:02:10.172465 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:02:10.172425 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd96f278_3477_457e_9a09_56fff149ba61.slice/crio-416f846e2ce191d78559c308adf8e3bb4980a2c4f891bba05abc507ed658297e WatchSource:0}: Error finding container 416f846e2ce191d78559c308adf8e3bb4980a2c4f891bba05abc507ed658297e: Status 404 returned error can't find the container with id 416f846e2ce191d78559c308adf8e3bb4980a2c4f891bba05abc507ed658297e Oct 30 22:02:10.174812 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.174783 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.174906 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.174850 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.174906 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.174878 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.223249 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.223221 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt"] Oct 30 22:02:10.225089 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:02:10.225064 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9935d58c_f632_459c_b66a_1f3843ac59b3.slice/crio-911a9e8489ab1315667999b196ff1d4168cb5adc4736f8330252fa2277eaae1d WatchSource:0}: Error finding container 911a9e8489ab1315667999b196ff1d4168cb5adc4736f8330252fa2277eaae1d: Status 404 returned error can't find the container with id 911a9e8489ab1315667999b196ff1d4168cb5adc4736f8330252fa2277eaae1d Oct 30 22:02:10.227875 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.227840 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.227954 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.227930 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.227994 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.227974 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.518910 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.518812 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" event={"ID":"9935d58c-f632-459c-b66a-1f3843ac59b3","Type":"ContainerStarted","Data":"c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318"} Oct 30 22:02:10.518910 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.518877 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" event={"ID":"9935d58c-f632-459c-b66a-1f3843ac59b3","Type":"ContainerStarted","Data":"911a9e8489ab1315667999b196ff1d4168cb5adc4736f8330252fa2277eaae1d"} Oct 30 22:02:10.520751 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.520718 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" event={"ID":"dd96f278-3477-457e-9a09-56fff149ba61","Type":"ContainerStarted","Data":"f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56"} Oct 30 22:02:10.520882 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.520780 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" event={"ID":"dd96f278-3477-457e-9a09-56fff149ba61","Type":"ContainerStarted","Data":"416f846e2ce191d78559c308adf8e3bb4980a2c4f891bba05abc507ed658297e"} Oct 30 22:02:10.646511 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.644458 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" podStartSLOduration=1.644436215 podStartE2EDuration="1.644436215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:10.638952956 +0000 UTC m=+2720.772326155" watchObservedRunningTime="2026-10-30 22:02:10.644436215 +0000 UTC m=+2720.777809416" Oct 30 22:02:10.708513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.703584 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:10.736800 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:10.736671 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:11.730513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:11.717601 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:02:11.730513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:11.717666 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 22:02:11.744650 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:11.744604 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:02:11.744803 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:11.744685 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 30 22:02:12.478445 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:02:12.478406 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:02:12.713524 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:12.713265 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:02:12.713524 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:12.713330 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 22:02:12.738570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:12.736533 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:02:12.738570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:12.736592 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 30 22:02:13.729515 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:13.707927 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:02:13.729515 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:13.707992 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 22:02:13.742512 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:13.739752 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:02:13.742512 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:13.739811 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 30 22:02:14.713512 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:14.709890 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:02:14.713512 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:14.709954 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 22:02:14.740907 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:14.740868 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:02:14.741202 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:14.741175 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 30 22:02:15.706936 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:15.706897 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 22:02:15.707631 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:15.707597 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 22:02:15.736919 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:15.736882 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 30 22:02:15.737221 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:15.737187 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 30 22:02:16.724041 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:16.724001 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:16.725057 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:16.725021 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:16.726364 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:16.726342 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:02:16.798982 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:16.798949 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:16.799315 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:16.799282 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:16.800182 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:16.800162 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:02:16.892636 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:16.892578 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" podStartSLOduration=7.892563695 podStartE2EDuration="7.892563695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:10.755705583 +0000 UTC m=+2720.889078782" watchObservedRunningTime="2026-10-30 22:02:16.892563695 +0000 UTC m=+2727.025936893" Oct 30 22:02:19.056389 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:19.056352 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846"] Oct 30 22:02:19.056895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:19.056590 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" containerID="cri-o://472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b" gracePeriod=30 Oct 30 22:02:21.351340 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.351303 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24"] Oct 30 22:02:21.355243 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.355218 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.411316 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.411282 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:21.457907 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.457875 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:02:21.462033 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.462005 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24"] Oct 30 22:02:21.523957 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.523915 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.524144 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.524102 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbqm\" (UniqueName: \"kubernetes.io/projected/921f5561-e6c0-41df-8269-9af6f5acbe32-kube-api-access-njbqm\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.524249 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.524141 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.625595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.625470 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbqm\" (UniqueName: \"kubernetes.io/projected/921f5561-e6c0-41df-8269-9af6f5acbe32-kube-api-access-njbqm\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.625595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.625545 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.625858 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.625684 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.626254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.626227 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.628192 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.628171 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.691748 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.691709 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbqm\" (UniqueName: \"kubernetes.io/projected/921f5561-e6c0-41df-8269-9af6f5acbe32-kube-api-access-njbqm\") pod \"maas-api-revoke-keys-e2e-mt-75c2acf6-74h24\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:21.969983 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:21.969894 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:22.348074 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:22.348043 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24"] Oct 30 22:02:22.349685 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:02:22.349656 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921f5561_e6c0_41df_8269_9af6f5acbe32.slice/crio-cf0741b1ad11f1cb4c3b4100c0d6ca06303c7b9e2c5be662a8b6774df1867c05 WatchSource:0}: Error finding container cf0741b1ad11f1cb4c3b4100c0d6ca06303c7b9e2c5be662a8b6774df1867c05: Status 404 returned error can't find the container with id cf0741b1ad11f1cb4c3b4100c0d6ca06303c7b9e2c5be662a8b6774df1867c05 Oct 30 22:02:22.628567 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:22.628535 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" event={"ID":"921f5561-e6c0-41df-8269-9af6f5acbe32","Type":"ContainerStarted","Data":"e80327752dfdfaa73033f622d445104ad5d02a3df55da025ebb936a63c3ea4b3"} Oct 30 22:02:22.629379 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:22.628585 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" event={"ID":"921f5561-e6c0-41df-8269-9af6f5acbe32","Type":"ContainerStarted","Data":"cf0741b1ad11f1cb4c3b4100c0d6ca06303c7b9e2c5be662a8b6774df1867c05"} Oct 30 22:02:22.742722 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:22.742668 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" podStartSLOduration=1.742654324 podStartE2EDuration="1.742654324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:22.740155171 +0000 UTC m=+2732.873528369" watchObservedRunningTime="2026-10-30 22:02:22.742654324 +0000 UTC m=+2732.876027521" Oct 30 22:02:23.636579 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:23.636547 4526 generic.go:356] "Generic (PLEG): container finished" podID="921f5561-e6c0-41df-8269-9af6f5acbe32" containerID="e80327752dfdfaa73033f622d445104ad5d02a3df55da025ebb936a63c3ea4b3" exitCode=0 Oct 30 22:02:23.637065 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:23.636637 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" event={"ID":"921f5561-e6c0-41df-8269-9af6f5acbe32","Type":"ContainerDied","Data":"e80327752dfdfaa73033f622d445104ad5d02a3df55da025ebb936a63c3ea4b3"} Oct 30 22:02:24.076764 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.076735 4526 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 22:02:24.076907 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.076785 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 22:02:24.335643 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.335584 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 22:02:24.360382 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360347 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-podinfo\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.360382 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360382 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istiod-ca-cert\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.360638 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360403 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-socket\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.360638 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360531 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-credential-socket\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.360638 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360586 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-certs\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.360638 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360624 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-data\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.360854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360661 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-token\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-token\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.360854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360711 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-kube-api-access-l6hc5\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-kube-api-access-l6hc5\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.360854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360744 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-socket" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:24.360854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360769 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-envoy\") pod \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\" (UID: \"61876d79-62d0-49d1-9f3a-8e78d0e1e67b\") " Oct 30 22:02:24.361054 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360893 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-certs" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:24.361054 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360918 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istiod-ca-cert" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:24.361054 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.360944 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-data" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:24.361054 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.361006 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-credential-socket" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:24.363021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.362927 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.363021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.362955 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.363021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.362972 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.363021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.362985 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.363021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.362999 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.363705 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.363683 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-podinfo" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:24.363846 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.363823 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-token" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:24.364076 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.364058 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-envoy" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:24.364076 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.364059 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-kube-api-access-l6hc5" pod "61876d79-62d0-49d1-9f3a-8e78d0e1e67b" (UID: "61876d79-62d0-49d1-9f3a-8e78d0e1e67b"). InnerVolumeSpecName "kube-api-access-l6hc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:24.464075 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.464033 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.464075 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.464069 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6hc5\" (UniqueName: \"kubernetes.io/projected/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-kube-api-access-l6hc5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.464075 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.464083 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.464320 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.464095 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61876d79-62d0-49d1-9f3a-8e78d0e1e67b-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.645963 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.645862 4526 generic.go:356] "Generic (PLEG): container finished" podID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerID="472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b" exitCode=0 Oct 30 22:02:24.645963 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.645914 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" Oct 30 22:02:24.645963 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.645944 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" event={"ID":"61876d79-62d0-49d1-9f3a-8e78d0e1e67b","Type":"ContainerDied","Data":"472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b"} Oct 30 22:02:24.646530 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.646003 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846" event={"ID":"61876d79-62d0-49d1-9f3a-8e78d0e1e67b","Type":"ContainerDied","Data":"e439145c0a0c5e7a474e1da4c96d85f8aea4d5bc62b430bdbd4dd98001152cf7"} Oct 30 22:02:24.646530 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.646025 4526 scope.go:122] "RemoveContainer" containerID="472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b" Oct 30 22:02:24.661248 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.661221 4526 scope.go:122] "RemoveContainer" containerID="68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee" Oct 30 22:02:24.670941 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.670914 4526 scope.go:122] "RemoveContainer" containerID="472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b" Oct 30 22:02:24.671284 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:02:24.671260 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b\": container with ID starting with 472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b not found: ID does not exist" containerID="472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b" Oct 30 22:02:24.671394 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.671293 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b"} err="failed to get container status \"472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b\": rpc error: code = NotFound desc = could not find container \"472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b\": container with ID starting with 472e0e164990ad323b80317e405bd7eadf5bf1793f567018def309b94c37ce0b not found: ID does not exist" Oct 30 22:02:24.671394 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.671312 4526 scope.go:122] "RemoveContainer" containerID="68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee" Oct 30 22:02:24.671711 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:02:24.671690 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee\": container with ID starting with 68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee not found: ID does not exist" containerID="68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee" Oct 30 22:02:24.671763 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.671722 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee"} err="failed to get container status \"68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee\": rpc error: code = NotFound desc = could not find container \"68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee\": container with ID starting with 68e41acb69541ac6799e9131e806cf257a527322ab322c45fd9516e34b516dee not found: ID does not exist" Oct 30 22:02:24.785628 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.785600 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:24.809334 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.809300 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846"] Oct 30 22:02:24.812895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.812868 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-99846"] Oct 30 22:02:24.868840 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.868806 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-xks-ca\") pod \"921f5561-e6c0-41df-8269-9af6f5acbe32\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " Oct 30 22:02:24.869030 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.868975 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921f5561-e6c0-41df-8269-9af6f5acbe32-kube-api-access-njbqm\" (UniqueName: \"kubernetes.io/projected/921f5561-e6c0-41df-8269-9af6f5acbe32-kube-api-access-njbqm\") pod \"921f5561-e6c0-41df-8269-9af6f5acbe32\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " Oct 30 22:02:24.869030 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.869011 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-openshift-ca\") pod \"921f5561-e6c0-41df-8269-9af6f5acbe32\" (UID: \"921f5561-e6c0-41df-8269-9af6f5acbe32\") " Oct 30 22:02:24.869421 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.869387 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-openshift-ca" pod "921f5561-e6c0-41df-8269-9af6f5acbe32" (UID: "921f5561-e6c0-41df-8269-9af6f5acbe32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:24.871158 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.871133 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-xks-ca" pod "921f5561-e6c0-41df-8269-9af6f5acbe32" (UID: "921f5561-e6c0-41df-8269-9af6f5acbe32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:24.871233 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.871164 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921f5561-e6c0-41df-8269-9af6f5acbe32-kube-api-access-njbqm" pod "921f5561-e6c0-41df-8269-9af6f5acbe32" (UID: "921f5561-e6c0-41df-8269-9af6f5acbe32"). InnerVolumeSpecName "kube-api-access-njbqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:24.971008 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.970919 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njbqm\" (UniqueName: \"kubernetes.io/projected/921f5561-e6c0-41df-8269-9af6f5acbe32-kube-api-access-njbqm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.971008 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.970951 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.971008 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:24.970962 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/921f5561-e6c0-41df-8269-9af6f5acbe32-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:25.654581 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:25.654550 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" Oct 30 22:02:25.655004 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:25.654549 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24" event={"ID":"921f5561-e6c0-41df-8269-9af6f5acbe32","Type":"ContainerDied","Data":"cf0741b1ad11f1cb4c3b4100c0d6ca06303c7b9e2c5be662a8b6774df1867c05"} Oct 30 22:02:25.655004 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:25.654674 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf0741b1ad11f1cb4c3b4100c0d6ca06303c7b9e2c5be662a8b6774df1867c05" Oct 30 22:02:25.985551 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:25.985466 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" path="/var/lib/kubelet/pods/61876d79-62d0-49d1-9f3a-8e78d0e1e67b/volumes" Oct 30 22:02:28.214941 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:28.214891 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podUID="a4207178a950303c256d4f333088d96f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 22:02:40.741716 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.741622 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2"] Oct 30 22:02:40.742288 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.742266 4526 cpu_manager.go:414] "containerMap: removing container" podUID="921f5561-e6c0-41df-8269-9af6f5acbe32" containerName="revoke-keys" Oct 30 22:02:40.742377 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.742290 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921f5561-e6c0-41df-8269-9af6f5acbe32" containerName="revoke-keys" Oct 30 22:02:40.742377 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.742306 4526 cpu_manager.go:414] "containerMap: removing container" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" Oct 30 22:02:40.742377 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.742314 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" Oct 30 22:02:40.742591 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.742505 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" Oct 30 22:02:40.742591 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.742522 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="921f5561-e6c0-41df-8269-9af6f5acbe32" containerName="revoke-keys" Oct 30 22:02:40.742591 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.742534 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" Oct 30 22:02:40.746944 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.746924 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:40.799094 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.799053 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:40.799320 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.799157 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:02:40.802846 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.802821 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2"] Oct 30 22:02:40.863353 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.863318 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64427\" (UniqueName: \"kubernetes.io/projected/2c1c0a1a-d924-4849-9844-d821d9cd3528-kube-api-access-64427\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:40.863565 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.863393 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:40.863565 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.863528 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:40.965127 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.965081 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64427\" (UniqueName: \"kubernetes.io/projected/2c1c0a1a-d924-4849-9844-d821d9cd3528-kube-api-access-64427\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:40.965317 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.965192 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:40.965317 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.965262 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:40.965994 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.965971 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:40.967929 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:40.967906 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:41.023050 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:41.022973 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64427\" (UniqueName: \"kubernetes.io/projected/2c1c0a1a-d924-4849-9844-d821d9cd3528-kube-api-access-64427\") pod \"maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:41.060422 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:41.060387 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:41.439749 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:41.439722 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2"] Oct 30 22:02:41.442427 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:02:41.442394 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1c0a1a_d924_4849_9844_d821d9cd3528.slice/crio-af912fdd8991129008b0ead07e33e7b8de9e9fffc0598a4a5a5c0d72c9cab848 WatchSource:0}: Error finding container af912fdd8991129008b0ead07e33e7b8de9e9fffc0598a4a5a5c0d72c9cab848: Status 404 returned error can't find the container with id af912fdd8991129008b0ead07e33e7b8de9e9fffc0598a4a5a5c0d72c9cab848 Oct 30 22:02:41.756551 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:41.756511 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" event={"ID":"2c1c0a1a-d924-4849-9844-d821d9cd3528","Type":"ContainerStarted","Data":"9ffdc701bcc424f0d0b940cd7a3ab40c785909d1d4c0abf912bb2008b0b86da8"} Oct 30 22:02:41.757015 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:41.756598 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" event={"ID":"2c1c0a1a-d924-4849-9844-d821d9cd3528","Type":"ContainerStarted","Data":"af912fdd8991129008b0ead07e33e7b8de9e9fffc0598a4a5a5c0d72c9cab848"} Oct 30 22:02:41.872142 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:41.871912 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" podStartSLOduration=1.8718643670000001 podStartE2EDuration="1.871864367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:41.871523082 +0000 UTC m=+2752.004896274" watchObservedRunningTime="2026-10-30 22:02:41.871864367 +0000 UTC m=+2752.005237566" Oct 30 22:02:42.478348 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:02:42.478310 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:02:42.764330 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:42.764296 4526 generic.go:356] "Generic (PLEG): container finished" podID="2c1c0a1a-d924-4849-9844-d821d9cd3528" containerID="9ffdc701bcc424f0d0b940cd7a3ab40c785909d1d4c0abf912bb2008b0b86da8" exitCode=0 Oct 30 22:02:42.764841 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:42.764367 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" event={"ID":"2c1c0a1a-d924-4849-9844-d821d9cd3528","Type":"ContainerDied","Data":"9ffdc701bcc424f0d0b940cd7a3ab40c785909d1d4c0abf912bb2008b0b86da8"} Oct 30 22:02:43.925953 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:43.925924 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:44.102846 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.102814 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-openshift-ca\") pod \"2c1c0a1a-d924-4849-9844-d821d9cd3528\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " Oct 30 22:02:44.103092 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.102918 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1c0a1a-d924-4849-9844-d821d9cd3528-kube-api-access-64427\" (UniqueName: \"kubernetes.io/projected/2c1c0a1a-d924-4849-9844-d821d9cd3528-kube-api-access-64427\") pod \"2c1c0a1a-d924-4849-9844-d821d9cd3528\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " Oct 30 22:02:44.103092 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.102987 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-xks-ca\") pod \"2c1c0a1a-d924-4849-9844-d821d9cd3528\" (UID: \"2c1c0a1a-d924-4849-9844-d821d9cd3528\") " Oct 30 22:02:44.103267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.103234 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-openshift-ca" pod "2c1c0a1a-d924-4849-9844-d821d9cd3528" (UID: "2c1c0a1a-d924-4849-9844-d821d9cd3528"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:44.103409 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.103387 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:44.105441 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.105415 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-xks-ca" pod "2c1c0a1a-d924-4849-9844-d821d9cd3528" (UID: "2c1c0a1a-d924-4849-9844-d821d9cd3528"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:44.105610 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.105456 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1c0a1a-d924-4849-9844-d821d9cd3528-kube-api-access-64427" pod "2c1c0a1a-d924-4849-9844-d821d9cd3528" (UID: "2c1c0a1a-d924-4849-9844-d821d9cd3528"). InnerVolumeSpecName "kube-api-access-64427". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:44.205107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.205075 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64427\" (UniqueName: \"kubernetes.io/projected/2c1c0a1a-d924-4849-9844-d821d9cd3528-kube-api-access-64427\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:44.205107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.205105 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1c0a1a-d924-4849-9844-d821d9cd3528-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:44.777315 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.777274 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" Oct 30 22:02:44.777592 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.777278 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2" event={"ID":"2c1c0a1a-d924-4849-9844-d821d9cd3528","Type":"ContainerDied","Data":"af912fdd8991129008b0ead07e33e7b8de9e9fffc0598a4a5a5c0d72c9cab848"} Oct 30 22:02:44.777592 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:44.777403 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af912fdd8991129008b0ead07e33e7b8de9e9fffc0598a4a5a5c0d72c9cab848" Oct 30 22:02:52.524085 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:52.524052 4526 scope.go:122] "RemoveContainer" containerID="f5a488409b4b0137d9c18cda3536050c69e68c15f09a0deed401e77994a57f45" Oct 30 22:02:52.533649 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:52.533627 4526 scope.go:122] "RemoveContainer" containerID="13eef02290b1ac82a1d817d773964c1c34b490b326afee98d806a25841120b89" Oct 30 22:02:52.542168 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:52.542145 4526 scope.go:122] "RemoveContainer" containerID="5995eb22d8e5fef291b28d4f965cffa88a994f122d1e29559cd5488d077274b3" Oct 30 22:02:52.551980 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:52.551955 4526 scope.go:122] "RemoveContainer" containerID="b0d196762493e495c6548925ce2c827e571497199e6bbab86fe28d30ea3ed0bd" Oct 30 22:02:52.561075 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:52.561051 4526 scope.go:122] "RemoveContainer" containerID="1260edd8529f3c823b82df0dd83c184718a5ebf6c6128d4ff83a2f3e21563627" Oct 30 22:02:52.571377 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:02:52.571350 4526 scope.go:122] "RemoveContainer" containerID="afa2bd5e6210e1149d76fcaf9d0a9d1699ec53a34e1dc58ce2b4880eb406f881" Oct 30 22:03:00.050919 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:00.050879 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24"] Oct 30 22:03:00.100355 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:00.100324 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-75c2acf6-74h24"] Oct 30 22:03:01.973236 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:01.973201 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj"] Oct 30 22:03:01.973683 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:01.973420 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" containerID="cri-o://47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54" gracePeriod=30 Oct 30 22:03:01.985566 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:01.985534 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921f5561-e6c0-41df-8269-9af6f5acbe32" path="/var/lib/kubelet/pods/921f5561-e6c0-41df-8269-9af6f5acbe32/volumes" Oct 30 22:03:02.685790 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:02.685747 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hqlmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:02.685963 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:02.685810 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:07.232168 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.232137 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 22:03:07.351733 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.351702 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5703ef72-0530-4cc8-929b-36a65f3b55a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5703ef72-0530-4cc8-929b-36a65f3b55a5-istiod-ca-cert\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.351901 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.351743 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-kube-api-access-rcpgg\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-kube-api-access-rcpgg\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.351901 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.351773 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-certs\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.351901 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.351832 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-data\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.351901 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.351862 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-podinfo\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.352120 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.351923 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-token\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-token\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.352120 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352012 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-credential-socket\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.352120 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352064 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-envoy\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.352288 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352118 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-socket\") pod \"5703ef72-0530-4cc8-929b-36a65f3b55a5\" (UID: \"5703ef72-0530-4cc8-929b-36a65f3b55a5\") " Oct 30 22:03:07.352288 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352145 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-certs" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:07.352288 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352196 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5703ef72-0530-4cc8-929b-36a65f3b55a5-istiod-ca-cert" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:07.352288 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352198 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-data" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:07.352527 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352298 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-credential-socket" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:07.352527 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352453 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-socket" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:07.352764 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352658 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5703ef72-0530-4cc8-929b-36a65f3b55a5-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.352764 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352688 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.352764 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352737 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.353093 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352850 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.353093 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.352889 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.354542 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.354515 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-token" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:07.354629 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.354545 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-kube-api-access-rcpgg" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "kube-api-access-rcpgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:07.354778 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.354754 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-envoy" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:07.354871 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.354788 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-podinfo" pod "5703ef72-0530-4cc8-929b-36a65f3b55a5" (UID: "5703ef72-0530-4cc8-929b-36a65f3b55a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:03:07.454194 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.454162 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcpgg\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-kube-api-access-rcpgg\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.454194 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.454191 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.454194 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.454201 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.454396 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.454210 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5703ef72-0530-4cc8-929b-36a65f3b55a5-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:07.918559 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.918524 4526 generic.go:356] "Generic (PLEG): container finished" podID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerID="47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54" exitCode=0 Oct 30 22:03:07.918704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.918574 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" Oct 30 22:03:07.918704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.918601 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" event={"ID":"5703ef72-0530-4cc8-929b-36a65f3b55a5","Type":"ContainerDied","Data":"47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54"} Oct 30 22:03:07.918704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.918653 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj" event={"ID":"5703ef72-0530-4cc8-929b-36a65f3b55a5","Type":"ContainerDied","Data":"2fb28d9b6f4d56771d91d8fa44ee3eb54d7bc738bf3440a990eb9a4b8a32ceee"} Oct 30 22:03:07.918704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.918676 4526 scope.go:122] "RemoveContainer" containerID="47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54" Oct 30 22:03:07.929026 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.929006 4526 scope.go:122] "RemoveContainer" containerID="47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54" Oct 30 22:03:07.929292 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:03:07.929272 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54\": container with ID starting with 47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54 not found: ID does not exist" containerID="47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54" Oct 30 22:03:07.929354 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:07.929304 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54"} err="failed to get container status \"47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54\": rpc error: code = NotFound desc = could not find container \"47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54\": container with ID starting with 47bccfe9ec5a2dba407c02969f8c2a298c6074e627ec5633b9c08bb35e926f54 not found: ID does not exist" Oct 30 22:03:08.132475 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:08.132439 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj"] Oct 30 22:03:08.182401 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:08.182310 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hqlmj"] Oct 30 22:03:08.797620 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:08.797584 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5"] Oct 30 22:03:08.798081 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:08.797820 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" podUID="37b158e4-91ec-4c81-8223-e8394d7deb1b" containerName="maas-api" containerID="cri-o://6588e3cc9c03c8ac05d379e7ff4d9289b6aea4dee90d740a39aece1425272c8b" gracePeriod=30 Oct 30 22:03:08.926158 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:08.926126 4526 generic.go:356] "Generic (PLEG): container finished" podID="37b158e4-91ec-4c81-8223-e8394d7deb1b" containerID="6588e3cc9c03c8ac05d379e7ff4d9289b6aea4dee90d740a39aece1425272c8b" exitCode=0 Oct 30 22:03:08.926331 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:08.926209 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" event={"ID":"37b158e4-91ec-4c81-8223-e8394d7deb1b","Type":"ContainerDied","Data":"6588e3cc9c03c8ac05d379e7ff4d9289b6aea4dee90d740a39aece1425272c8b"} Oct 30 22:03:09.073047 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.073016 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 22:03:09.175784 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.175743 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-maas-api-tls\") pod \"37b158e4-91ec-4c81-8223-e8394d7deb1b\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " Oct 30 22:03:09.176530 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.176505 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b158e4-91ec-4c81-8223-e8394d7deb1b-kube-api-access-97q62\" (UniqueName: \"kubernetes.io/projected/37b158e4-91ec-4c81-8223-e8394d7deb1b-kube-api-access-97q62\") pod \"37b158e4-91ec-4c81-8223-e8394d7deb1b\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " Oct 30 22:03:09.176855 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.176817 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b158e4-91ec-4c81-8223-e8394d7deb1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b158e4-91ec-4c81-8223-e8394d7deb1b-openshift-service-ca\") pod \"37b158e4-91ec-4c81-8223-e8394d7deb1b\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " Oct 30 22:03:09.177342 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.177191 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b158e4-91ec-4c81-8223-e8394d7deb1b-openshift-service-ca" pod "37b158e4-91ec-4c81-8223-e8394d7deb1b" (UID: "37b158e4-91ec-4c81-8223-e8394d7deb1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:09.181975 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.177397 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-metrics-certs\") pod \"37b158e4-91ec-4c81-8223-e8394d7deb1b\" (UID: \"37b158e4-91ec-4c81-8223-e8394d7deb1b\") " Oct 30 22:03:09.181975 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.178283 4526 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37b158e4-91ec-4c81-8223-e8394d7deb1b-openshift-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:09.181975 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.181249 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-metrics-certs" pod "37b158e4-91ec-4c81-8223-e8394d7deb1b" (UID: "37b158e4-91ec-4c81-8223-e8394d7deb1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:09.181975 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.181544 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b158e4-91ec-4c81-8223-e8394d7deb1b-kube-api-access-97q62" pod "37b158e4-91ec-4c81-8223-e8394d7deb1b" (UID: "37b158e4-91ec-4c81-8223-e8394d7deb1b"). InnerVolumeSpecName "kube-api-access-97q62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:09.183077 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.183053 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-maas-api-tls" pod "37b158e4-91ec-4c81-8223-e8394d7deb1b" (UID: "37b158e4-91ec-4c81-8223-e8394d7deb1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:09.279384 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.279347 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97q62\" (UniqueName: \"kubernetes.io/projected/37b158e4-91ec-4c81-8223-e8394d7deb1b-kube-api-access-97q62\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:09.279384 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.279378 4526 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-metrics-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:09.279384 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.279389 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37b158e4-91ec-4c81-8223-e8394d7deb1b-maas-api-tls\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:09.934500 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.934461 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" event={"ID":"37b158e4-91ec-4c81-8223-e8394d7deb1b","Type":"ContainerDied","Data":"652f3d1b7750d50efd3dc12a61f266244492321a36bd31ff0af5fb29b1150a12"} Oct 30 22:03:09.934996 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.934520 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5" Oct 30 22:03:09.934996 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.934547 4526 scope.go:122] "RemoveContainer" containerID="6588e3cc9c03c8ac05d379e7ff4d9289b6aea4dee90d740a39aece1425272c8b" Oct 30 22:03:09.984680 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:09.984653 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" path="/var/lib/kubelet/pods/5703ef72-0530-4cc8-929b-36a65f3b55a5/volumes" Oct 30 22:03:10.100866 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:10.100829 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5"] Oct 30 22:03:10.150504 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:10.150454 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e9aa8e-9897494c9-d5zx5"] Oct 30 22:03:10.882054 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:10.882018 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2"] Oct 30 22:03:10.931700 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:10.931666 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e9aa8e-kjfd2"] Oct 30 22:03:11.985140 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:11.985106 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1c0a1a-d924-4849-9844-d821d9cd3528" path="/var/lib/kubelet/pods/2c1c0a1a-d924-4849-9844-d821d9cd3528/volumes" Oct 30 22:03:11.985625 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:11.985609 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b158e4-91ec-4c81-8223-e8394d7deb1b" path="/var/lib/kubelet/pods/37b158e4-91ec-4c81-8223-e8394d7deb1b/volumes" Oct 30 22:03:12.477663 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:03:12.477626 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:03:25.493522 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:25.493471 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4"] Oct 30 22:03:28.586423 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586387 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv"] Oct 30 22:03:28.586884 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586854 4526 cpu_manager.go:414] "containerMap: removing container" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" Oct 30 22:03:28.586884 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586865 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61876d79-62d0-49d1-9f3a-8e78d0e1e67b" containerName="istio-proxy" Oct 30 22:03:28.586884 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586878 4526 cpu_manager.go:414] "containerMap: removing container" podUID="37b158e4-91ec-4c81-8223-e8394d7deb1b" containerName="maas-api" Oct 30 22:03:28.586884 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586884 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b158e4-91ec-4c81-8223-e8394d7deb1b" containerName="maas-api" Oct 30 22:03:28.587045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586893 4526 cpu_manager.go:414] "containerMap: removing container" podUID="2c1c0a1a-d924-4849-9844-d821d9cd3528" containerName="revoke-keys" Oct 30 22:03:28.587045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586898 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1c0a1a-d924-4849-9844-d821d9cd3528" containerName="revoke-keys" Oct 30 22:03:28.587045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586920 4526 cpu_manager.go:414] "containerMap: removing container" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" Oct 30 22:03:28.587045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586925 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" Oct 30 22:03:28.587045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.586992 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b158e4-91ec-4c81-8223-e8394d7deb1b" containerName="maas-api" Oct 30 22:03:28.587045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.587000 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1c0a1a-d924-4849-9844-d821d9cd3528" containerName="revoke-keys" Oct 30 22:03:28.587045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.587010 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="5703ef72-0530-4cc8-929b-36a65f3b55a5" containerName="istio-proxy" Oct 30 22:03:28.590214 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.590193 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.646350 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.646303 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:28.684989 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.684952 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbvw\" (UniqueName: \"kubernetes.io/projected/891751b2-2808-4654-9d5d-8649d780983d-kube-api-access-xnbvw\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.685201 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.685084 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891751b2-2808-4654-9d5d-8649d780983d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.685201 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.685117 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891751b2-2808-4654-9d5d-8649d780983d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.692625 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.692588 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv"] Oct 30 22:03:28.695823 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.695792 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:28.695991 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.695843 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:03:28.746897 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.746859 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:28.786975 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.786934 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbvw\" (UniqueName: \"kubernetes.io/projected/891751b2-2808-4654-9d5d-8649d780983d-kube-api-access-xnbvw\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.787177 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.787055 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891751b2-2808-4654-9d5d-8649d780983d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.787177 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.787087 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891751b2-2808-4654-9d5d-8649d780983d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.787886 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.787860 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891751b2-2808-4654-9d5d-8649d780983d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.789808 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.789784 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891751b2-2808-4654-9d5d-8649d780983d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.844969 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.844892 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbvw\" (UniqueName: \"kubernetes.io/projected/891751b2-2808-4654-9d5d-8649d780983d-kube-api-access-xnbvw\") pod \"maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:28.903819 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:28.903772 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:29.301908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:29.301878 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv"] Oct 30 22:03:29.304368 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:03:29.304323 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891751b2_2808_4654_9d5d_8649d780983d.slice/crio-18833fd5f663ea8b7b26a8b0efa199c3d47553a326d6f3b28c5b4b8e9433951c WatchSource:0}: Error finding container 18833fd5f663ea8b7b26a8b0efa199c3d47553a326d6f3b28c5b4b8e9433951c: Status 404 returned error can't find the container with id 18833fd5f663ea8b7b26a8b0efa199c3d47553a326d6f3b28c5b4b8e9433951c Oct 30 22:03:30.050941 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:30.050903 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" event={"ID":"891751b2-2808-4654-9d5d-8649d780983d","Type":"ContainerStarted","Data":"da651ef9d1f7d6aea1b97461230e298aa924d935bd801701ffa2a39ee3c8ad32"} Oct 30 22:03:30.051359 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:30.050961 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" event={"ID":"891751b2-2808-4654-9d5d-8649d780983d","Type":"ContainerStarted","Data":"18833fd5f663ea8b7b26a8b0efa199c3d47553a326d6f3b28c5b4b8e9433951c"} Oct 30 22:03:30.170737 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:30.170679 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" podStartSLOduration=2.170661014 podStartE2EDuration="2.170661014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:30.167735747 +0000 UTC m=+2800.301108947" watchObservedRunningTime="2026-10-30 22:03:30.170661014 +0000 UTC m=+2800.304034251" Oct 30 22:03:31.058764 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:31.058725 4526 generic.go:356] "Generic (PLEG): container finished" podID="891751b2-2808-4654-9d5d-8649d780983d" containerID="da651ef9d1f7d6aea1b97461230e298aa924d935bd801701ffa2a39ee3c8ad32" exitCode=0 Oct 30 22:03:31.059309 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:31.058811 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" event={"ID":"891751b2-2808-4654-9d5d-8649d780983d","Type":"ContainerDied","Data":"da651ef9d1f7d6aea1b97461230e298aa924d935bd801701ffa2a39ee3c8ad32"} Oct 30 22:03:32.204376 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.204348 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:32.324048 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.323969 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/891751b2-2808-4654-9d5d-8649d780983d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891751b2-2808-4654-9d5d-8649d780983d-maas-api-xks-ca\") pod \"891751b2-2808-4654-9d5d-8649d780983d\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " Oct 30 22:03:32.324232 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.324074 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891751b2-2808-4654-9d5d-8649d780983d-kube-api-access-xnbvw\" (UniqueName: \"kubernetes.io/projected/891751b2-2808-4654-9d5d-8649d780983d-kube-api-access-xnbvw\") pod \"891751b2-2808-4654-9d5d-8649d780983d\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " Oct 30 22:03:32.324232 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.324102 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891751b2-2808-4654-9d5d-8649d780983d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891751b2-2808-4654-9d5d-8649d780983d-maas-api-openshift-ca\") pod \"891751b2-2808-4654-9d5d-8649d780983d\" (UID: \"891751b2-2808-4654-9d5d-8649d780983d\") " Oct 30 22:03:32.324598 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.324567 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891751b2-2808-4654-9d5d-8649d780983d-maas-api-openshift-ca" pod "891751b2-2808-4654-9d5d-8649d780983d" (UID: "891751b2-2808-4654-9d5d-8649d780983d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:32.326647 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.326618 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891751b2-2808-4654-9d5d-8649d780983d-kube-api-access-xnbvw" pod "891751b2-2808-4654-9d5d-8649d780983d" (UID: "891751b2-2808-4654-9d5d-8649d780983d"). InnerVolumeSpecName "kube-api-access-xnbvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:32.326779 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.326649 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891751b2-2808-4654-9d5d-8649d780983d-maas-api-xks-ca" pod "891751b2-2808-4654-9d5d-8649d780983d" (UID: "891751b2-2808-4654-9d5d-8649d780983d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:32.425605 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.425565 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/891751b2-2808-4654-9d5d-8649d780983d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:32.425605 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.425599 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbvw\" (UniqueName: \"kubernetes.io/projected/891751b2-2808-4654-9d5d-8649d780983d-kube-api-access-xnbvw\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:32.425831 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:32.425616 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/891751b2-2808-4654-9d5d-8649d780983d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:33.071689 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:33.071655 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" Oct 30 22:03:33.071886 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:33.071655 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv" event={"ID":"891751b2-2808-4654-9d5d-8649d780983d","Type":"ContainerDied","Data":"18833fd5f663ea8b7b26a8b0efa199c3d47553a326d6f3b28c5b4b8e9433951c"} Oct 30 22:03:33.071886 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:33.071786 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18833fd5f663ea8b7b26a8b0efa199c3d47553a326d6f3b28c5b4b8e9433951c" Oct 30 22:03:40.323034 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.323001 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8"] Oct 30 22:03:40.514858 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.514804 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" podUID="7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" containerName="main" containerID="cri-o://29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6" gracePeriod=45 Oct 30 22:03:40.577162 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.577098 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" podUID="7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.231:8000/ready\": dial tcp 10.133.0.231:8000: connect: connection refused" Oct 30 22:03:40.788854 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.788828 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 22:03:40.919072 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919001 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tls-certs\") pod \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " Oct 30 22:03:40.919254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919090 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-kube-api-access-dv4zz\" (UniqueName: \"kubernetes.io/projected/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-kube-api-access-dv4zz\") pod \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " Oct 30 22:03:40.919254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919156 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-model-cache\") pod \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " Oct 30 22:03:40.919254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919197 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-home\") pod \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " Oct 30 22:03:40.919254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919241 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tmp-dir\") pod \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " Oct 30 22:03:40.919537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919319 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-dshm\") pod \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\" (UID: \"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f\") " Oct 30 22:03:40.919898 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919560 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-home" pod "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" (UID: "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:40.919898 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919649 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tmp-dir" pod "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" (UID: "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:40.919898 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.919706 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-model-cache" pod "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" (UID: "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:40.920305 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.920001 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:40.920305 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.920025 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:40.920305 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.920039 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:40.921715 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.921690 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tls-certs" pod "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" (UID: "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:40.921911 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.921895 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-kube-api-access-dv4zz" pod "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" (UID: "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f"). InnerVolumeSpecName "kube-api-access-dv4zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:40.921991 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:40.921925 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-dshm" pod "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" (UID: "7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:41.021513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.021460 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv4zz\" (UniqueName: \"kubernetes.io/projected/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-kube-api-access-dv4zz\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:41.021678 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.021519 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:41.021678 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.021535 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:41.118132 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.118099 4526 generic.go:356] "Generic (PLEG): container finished" podID="7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" containerID="29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6" exitCode=0 Oct 30 22:03:41.118315 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.118148 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" Oct 30 22:03:41.118315 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.118184 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" event={"ID":"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f","Type":"ContainerDied","Data":"29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6"} Oct 30 22:03:41.118315 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.118230 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4" event={"ID":"7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f","Type":"ContainerDied","Data":"e8ab04cacbd93b871a0c0f25a4c21102e3ffa15eb5ca5ed7690c04d626fef81a"} Oct 30 22:03:41.118315 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.118249 4526 scope.go:122] "RemoveContainer" containerID="29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6" Oct 30 22:03:41.128843 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.128823 4526 scope.go:122] "RemoveContainer" containerID="29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6" Oct 30 22:03:41.129121 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:03:41.129103 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6\": container with ID starting with 29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6 not found: ID does not exist" containerID="29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6" Oct 30 22:03:41.129174 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.129128 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6"} err="failed to get container status \"29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6\": rpc error: code = NotFound desc = could not find container \"29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6\": container with ID starting with 29ab9a947a8d8c6c2ade49d33db76e0ba6506d7b41bb8ae2c4c7a89b8c7f0fe6 not found: ID does not exist" Oct 30 22:03:41.292410 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.292380 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4"] Oct 30 22:03:41.303980 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.303948 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-75c2acf6/e2e-lifecycle-model-75c2acf6-kserve-649cddf597-8hfv4"] Oct 30 22:03:41.984704 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:41.984665 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" path="/var/lib/kubelet/pods/7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f/volumes" Oct 30 22:03:42.478079 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:03:42.478045 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:03:45.323467 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:45.323400 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" podUID="dd2b81c7-e253-43bd-bdab-119d0f2abc1c" containerName="payload-pre-processing" containerID="cri-o://08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff" gracePeriod=25 Oct 30 22:03:50.981130 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:50.981096 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" podUID="dd2b81c7-e253-43bd-bdab-119d0f2abc1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:03:52.621671 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:52.621637 4526 scope.go:122] "RemoveContainer" containerID="ad957717421aff33112cb51f063e69326f50a4fa3fb19fc3604f8121ffc56bb2" Oct 30 22:03:52.630461 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:52.630429 4526 scope.go:122] "RemoveContainer" containerID="3f9d2c003da9722d149d35d71b6953cf88ff805e18ca9ba8dfba298cac1728c7" Oct 30 22:03:52.638254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:52.638235 4526 scope.go:122] "RemoveContainer" containerID="8cbc2430a598fff32ad44d9bb05c1c14b177e6e706d088ac0931dbfc450508af" Oct 30 22:03:52.645919 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:52.645903 4526 scope.go:122] "RemoveContainer" containerID="b6ab07c63baff310e4415fa38b4ae07cde443964afd6bbfa8d388a93e269251e" Oct 30 22:03:55.947352 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:55.947312 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv"] Oct 30 22:03:55.949730 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:55.949707 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-dbe6da-tw7cv"] Oct 30 22:03:55.985320 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:03:55.985290 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891751b2-2808-4654-9d5d-8649d780983d" path="/var/lib/kubelet/pods/891751b2-2808-4654-9d5d-8649d780983d/volumes" Oct 30 22:04:00.981919 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:00.981879 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" podUID="dd2b81c7-e253-43bd-bdab-119d0f2abc1c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:03.658044 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:03.658012 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh"] Oct 30 22:04:03.658517 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:03.658233 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" containerID="cri-o://d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22" gracePeriod=30 Oct 30 22:04:03.768363 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:03.768317 4526 patch_prober.go:29] interesting pod/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:04:03.768563 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:03.768372 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:04:05.587128 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:05.587056 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 22:04:05.688321 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:05.688289 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-plugins-config-volume\") pod \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\" (UID: \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\") " Oct 30 22:04:05.688565 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:05.688369 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-kube-api-access-x54s2\" (UniqueName: \"kubernetes.io/projected/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-kube-api-access-x54s2\") pod \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\" (UID: \"dd2b81c7-e253-43bd-bdab-119d0f2abc1c\") " Oct 30 22:04:05.688825 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:05.688784 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-plugins-config-volume" pod "dd2b81c7-e253-43bd-bdab-119d0f2abc1c" (UID: "dd2b81c7-e253-43bd-bdab-119d0f2abc1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:05.691115 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:05.691085 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-kube-api-access-x54s2" pod "dd2b81c7-e253-43bd-bdab-119d0f2abc1c" (UID: "dd2b81c7-e253-43bd-bdab-119d0f2abc1c"). InnerVolumeSpecName "kube-api-access-x54s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:05.789551 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:05.789515 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:05.789551 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:05.789547 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x54s2\" (UniqueName: \"kubernetes.io/projected/dd2b81c7-e253-43bd-bdab-119d0f2abc1c-kube-api-access-x54s2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:06.264351 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.264270 4526 generic.go:356] "Generic (PLEG): container finished" podID="dd2b81c7-e253-43bd-bdab-119d0f2abc1c" containerID="08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff" exitCode=0 Oct 30 22:04:06.264351 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.264338 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" event={"ID":"dd2b81c7-e253-43bd-bdab-119d0f2abc1c","Type":"ContainerDied","Data":"08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff"} Oct 30 22:04:06.264586 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.264370 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" Oct 30 22:04:06.264586 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.264394 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8" event={"ID":"dd2b81c7-e253-43bd-bdab-119d0f2abc1c","Type":"ContainerDied","Data":"e2008a6c6193703072216d0994636c8a19ef3f301bab759a9bb427a224df3ffd"} Oct 30 22:04:06.264586 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.264424 4526 scope.go:122] "RemoveContainer" containerID="08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff" Oct 30 22:04:06.274071 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.274047 4526 scope.go:122] "RemoveContainer" containerID="08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff" Oct 30 22:04:06.274385 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:06.274365 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff\": container with ID starting with 08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff not found: ID does not exist" containerID="08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff" Oct 30 22:04:06.274434 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.274396 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff"} err="failed to get container status \"08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff\": rpc error: code = NotFound desc = could not find container \"08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff\": container with ID starting with 08db1c88582197963e43312645eb9ef372d3fc8b7db2642f01982ada3a04caff not found: ID does not exist" Oct 30 22:04:06.427881 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.427845 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8"] Oct 30 22:04:06.477357 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:06.477323 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dbe6da-58db7f7b59-chhr8"] Oct 30 22:04:07.984771 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:07.984740 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2b81c7-e253-43bd-bdab-119d0f2abc1c" path="/var/lib/kubelet/pods/dd2b81c7-e253-43bd-bdab-119d0f2abc1c/volumes" Oct 30 22:04:08.909986 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:08.909956 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 22:04:09.023475 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023386 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-socket\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.023475 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023467 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-podinfo\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023539 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-kube-api-access-kzw2z\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-kube-api-access-kzw2z\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023580 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-certs\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023655 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-data\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023698 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-envoy\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023750 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-token\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-token\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023755 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-socket" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023784 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-credential-socket\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023832 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istiod-ca-cert\") pod \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\" (UID: \"351cddf1-72d9-404d-b1e8-d1592a4bffe8\") " Oct 30 22:04:09.024035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.023914 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-certs" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.024560 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.024257 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-data" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.024560 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.024388 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-credential-socket" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.024560 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.024523 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istiod-ca-cert" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:09.024742 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.024642 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.024742 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.024671 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.024742 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.024687 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.024742 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.024699 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.024742 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.024713 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.026324 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.026300 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-kube-api-access-kzw2z" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "kube-api-access-kzw2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:09.026764 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.026746 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-podinfo" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:09.026865 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.026768 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-token" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:09.026960 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.026944 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-envoy" pod "351cddf1-72d9-404d-b1e8-d1592a4bffe8" (UID: "351cddf1-72d9-404d-b1e8-d1592a4bffe8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:09.126323 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.126285 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.126560 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.126337 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.126560 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.126348 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351cddf1-72d9-404d-b1e8-d1592a4bffe8-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.126560 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.126372 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzw2z\" (UniqueName: \"kubernetes.io/projected/351cddf1-72d9-404d-b1e8-d1592a4bffe8-kube-api-access-kzw2z\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:09.290983 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.286262 4526 generic.go:356] "Generic (PLEG): container finished" podID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerID="d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22" exitCode=0 Oct 30 22:04:09.290983 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.286432 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" event={"ID":"351cddf1-72d9-404d-b1e8-d1592a4bffe8","Type":"ContainerDied","Data":"d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22"} Oct 30 22:04:09.290983 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.286574 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" event={"ID":"351cddf1-72d9-404d-b1e8-d1592a4bffe8","Type":"ContainerDied","Data":"67b5089cb6202a531a08f08636e29a32c9b7de42ab9bc70f7a177116ad778880"} Oct 30 22:04:09.290983 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.286616 4526 scope.go:122] "RemoveContainer" containerID="d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22" Oct 30 22:04:09.290983 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.286815 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh" Oct 30 22:04:09.301638 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.301617 4526 scope.go:122] "RemoveContainer" containerID="fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d" Oct 30 22:04:09.310500 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.310470 4526 scope.go:122] "RemoveContainer" containerID="d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22" Oct 30 22:04:09.310758 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:09.310739 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22\": container with ID starting with d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22 not found: ID does not exist" containerID="d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22" Oct 30 22:04:09.310810 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.310767 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22"} err="failed to get container status \"d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22\": rpc error: code = NotFound desc = could not find container \"d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22\": container with ID starting with d8d4fce857a561ad546b4179cef50f1d5ee75bce83b4b29e49d0f05c7a9a4d22 not found: ID does not exist" Oct 30 22:04:09.310810 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.310784 4526 scope.go:122] "RemoveContainer" containerID="fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d" Oct 30 22:04:09.311001 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:09.310984 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d\": container with ID starting with fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d not found: ID does not exist" containerID="fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d" Oct 30 22:04:09.311043 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.311007 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d"} err="failed to get container status \"fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d\": rpc error: code = NotFound desc = could not find container \"fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d\": container with ID starting with fca84fe8d4088fc0eee412a9a372ca602452da8d0b926dc6e14c4524edc33d4d not found: ID does not exist" Oct 30 22:04:09.410963 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.410936 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h"] Oct 30 22:04:09.411372 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.411341 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" containerID="cri-o://f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56" gracePeriod=30 Oct 30 22:04:09.414138 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.414103 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c"] Oct 30 22:04:09.414360 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.414329 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" containerID="cri-o://b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8" gracePeriod=30 Oct 30 22:04:09.436132 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.436096 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt"] Oct 30 22:04:09.436319 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.436301 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" containerID="cri-o://c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318" gracePeriod=30 Oct 30 22:04:09.454673 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.454639 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh"] Oct 30 22:04:09.459219 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.459186 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dbe6da-openshift-default-6d448cf4fc-6zrwh"] Oct 30 22:04:09.986000 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:09.985968 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" path="/var/lib/kubelet/pods/351cddf1-72d9-404d-b1e8-d1592a4bffe8/volumes" Oct 30 22:04:10.411260 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411225 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv"] Oct 30 22:04:10.411694 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411680 4526 cpu_manager.go:414] "containerMap: removing container" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" Oct 30 22:04:10.411741 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411696 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" Oct 30 22:04:10.411741 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411710 4526 cpu_manager.go:414] "containerMap: removing container" podUID="891751b2-2808-4654-9d5d-8649d780983d" containerName="revoke-keys" Oct 30 22:04:10.411741 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411715 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891751b2-2808-4654-9d5d-8649d780983d" containerName="revoke-keys" Oct 30 22:04:10.411741 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411737 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" containerName="main" Oct 30 22:04:10.411741 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411742 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" containerName="main" Oct 30 22:04:10.411918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411754 4526 cpu_manager.go:414] "containerMap: removing container" podUID="dd2b81c7-e253-43bd-bdab-119d0f2abc1c" containerName="payload-pre-processing" Oct 30 22:04:10.411918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411759 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2b81c7-e253-43bd-bdab-119d0f2abc1c" containerName="payload-pre-processing" Oct 30 22:04:10.411918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411850 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" Oct 30 22:04:10.411918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411859 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="891751b2-2808-4654-9d5d-8649d780983d" containerName="revoke-keys" Oct 30 22:04:10.411918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411867 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" Oct 30 22:04:10.411918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411872 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe0ccf4-b9cc-4948-92fd-5f2fedc7176f" containerName="main" Oct 30 22:04:10.411918 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.411878 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2b81c7-e253-43bd-bdab-119d0f2abc1c" containerName="payload-pre-processing" Oct 30 22:04:10.416441 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.416418 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.468950 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.468801 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:10.468950 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.468852 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:10.468950 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.468918 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:10.469473 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.468962 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:04:10.472084 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.472063 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv"] Oct 30 22:04:10.541725 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.541691 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.542016 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.541807 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.542016 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.541840 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmv7c\" (UniqueName: \"kubernetes.io/projected/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-kube-api-access-qmv7c\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.642460 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.642424 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.642653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.642531 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.642653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.642561 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmv7c\" (UniqueName: \"kubernetes.io/projected/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-kube-api-access-qmv7c\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.643210 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.643148 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.644975 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.644953 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.699397 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.699333 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmv7c\" (UniqueName: \"kubernetes.io/projected/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-kube-api-access-qmv7c\") pod \"maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:10.729322 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:10.729293 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:11.104926 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:11.104897 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv"] Oct 30 22:04:11.107294 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:04:11.107251 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d94dc2f_5efb_4b3b_a869_8e415fdcac49.slice/crio-c477330c5d9a49a7e40237691d9d1dcc1741c90b057133f5e7f0ca45d93af300 WatchSource:0}: Error finding container c477330c5d9a49a7e40237691d9d1dcc1741c90b057133f5e7f0ca45d93af300: Status 404 returned error can't find the container with id c477330c5d9a49a7e40237691d9d1dcc1741c90b057133f5e7f0ca45d93af300 Oct 30 22:04:11.300775 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:11.300739 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" event={"ID":"7d94dc2f-5efb-4b3b-a869-8e415fdcac49","Type":"ContainerStarted","Data":"c477330c5d9a49a7e40237691d9d1dcc1741c90b057133f5e7f0ca45d93af300"} Oct 30 22:04:12.308278 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:12.308239 4526 generic.go:356] "Generic (PLEG): container finished" podID="7d94dc2f-5efb-4b3b-a869-8e415fdcac49" containerID="86f05b21391e9fbe86a444bfd8d3cad3ec17f8c9f81aba67212125dafadc66e9" exitCode=0 Oct 30 22:04:12.308729 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:12.308284 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" event={"ID":"7d94dc2f-5efb-4b3b-a869-8e415fdcac49","Type":"ContainerDied","Data":"86f05b21391e9fbe86a444bfd8d3cad3ec17f8c9f81aba67212125dafadc66e9"} Oct 30 22:04:12.442385 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:12.442346 4526 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:04:12.442595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:12.442401 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:04:12.479149 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:12.479107 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:04:13.461834 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.461804 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:13.575108 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.574977 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-kube-api-access-qmv7c\" (UniqueName: \"kubernetes.io/projected/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-kube-api-access-qmv7c\") pod \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " Oct 30 22:04:13.575108 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.575089 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-xks-ca\") pod \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " Oct 30 22:04:13.575375 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.575140 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-openshift-ca\") pod \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\" (UID: \"7d94dc2f-5efb-4b3b-a869-8e415fdcac49\") " Oct 30 22:04:13.577134 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.575614 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-openshift-ca" pod "7d94dc2f-5efb-4b3b-a869-8e415fdcac49" (UID: "7d94dc2f-5efb-4b3b-a869-8e415fdcac49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:13.577713 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.577688 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-kube-api-access-qmv7c" pod "7d94dc2f-5efb-4b3b-a869-8e415fdcac49" (UID: "7d94dc2f-5efb-4b3b-a869-8e415fdcac49"). InnerVolumeSpecName "kube-api-access-qmv7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:13.577943 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.577908 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-xks-ca" pod "7d94dc2f-5efb-4b3b-a869-8e415fdcac49" (UID: "7d94dc2f-5efb-4b3b-a869-8e415fdcac49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:13.677012 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.676968 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:13.677012 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.677005 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:13.677012 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:13.677016 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmv7c\" (UniqueName: \"kubernetes.io/projected/7d94dc2f-5efb-4b3b-a869-8e415fdcac49-kube-api-access-qmv7c\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.321979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.321950 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" Oct 30 22:04:14.322161 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.321944 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv" event={"ID":"7d94dc2f-5efb-4b3b-a869-8e415fdcac49","Type":"ContainerDied","Data":"c477330c5d9a49a7e40237691d9d1dcc1741c90b057133f5e7f0ca45d93af300"} Oct 30 22:04:14.322161 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.322073 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c477330c5d9a49a7e40237691d9d1dcc1741c90b057133f5e7f0ca45d93af300" Oct 30 22:04:14.616199 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.616164 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg"] Oct 30 22:04:14.617240 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.617218 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7d94dc2f-5efb-4b3b-a869-8e415fdcac49" containerName="revoke-keys" Oct 30 22:04:14.617364 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.617354 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d94dc2f-5efb-4b3b-a869-8e415fdcac49" containerName="revoke-keys" Oct 30 22:04:14.617428 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.617421 4526 cpu_manager.go:414] "containerMap: removing container" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" Oct 30 22:04:14.617507 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.617479 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351cddf1-72d9-404d-b1e8-d1592a4bffe8" containerName="istio-proxy" Oct 30 22:04:14.617653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.617634 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d94dc2f-5efb-4b3b-a869-8e415fdcac49" containerName="revoke-keys" Oct 30 22:04:14.621422 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.621402 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.673384 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.673342 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:14.673384 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.673352 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:14.673596 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.673394 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:14.673596 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.673506 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:04:14.693254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.693218 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldff8\" (UniqueName: \"kubernetes.io/projected/0b1c6200-5dd2-437c-b3b3-18b919ef7496-kube-api-access-ldff8\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.693426 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.693286 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.693538 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.693517 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.710192 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.710163 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg"] Oct 30 22:04:14.730721 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.730693 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 22:04:14.793043 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.793008 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:04:14.794176 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.794150 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-data\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.794310 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.794193 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-token\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-token\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.794310 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.794218 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-kube-api-access-5xcqp\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-kube-api-access-5xcqp\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.794504 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.794465 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-data" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.794647 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.794610 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-envoy\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.794767 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.794673 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-socket\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.794767 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.794749 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-credential-socket\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.794892 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.794795 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-certs\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.795076 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.795045 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90bb6a42-f6f9-47be-a5cc-ede068832de1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb6a42-f6f9-47be-a5cc-ede068832de1-istiod-ca-cert\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.795136 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.795083 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-podinfo\") pod \"90bb6a42-f6f9-47be-a5cc-ede068832de1\" (UID: \"90bb6a42-f6f9-47be-a5cc-ede068832de1\") " Oct 30 22:04:14.795136 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.795108 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:04:14.795406 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.795320 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-credential-socket" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.795584 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.795563 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-certs" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.795811 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.795783 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bb6a42-f6f9-47be-a5cc-ede068832de1-istiod-ca-cert" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:14.795953 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.795932 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-socket" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.796051 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796030 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldff8\" (UniqueName: \"kubernetes.io/projected/0b1c6200-5dd2-437c-b3b3-18b919ef7496-kube-api-access-ldff8\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.796119 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796095 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.796232 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796217 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.796326 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796312 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bb6a42-f6f9-47be-a5cc-ede068832de1-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.796388 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796331 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.796388 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796345 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.796388 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796360 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.796388 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796373 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.796972 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.796927 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-kube-api-access-5xcqp" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "kube-api-access-5xcqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.797224 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.797207 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.797389 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.797370 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-token" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.797682 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.797660 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-envoy" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.798274 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.798254 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-podinfo" pod "90bb6a42-f6f9-47be-a5cc-ede068832de1" (UID: "90bb6a42-f6f9-47be-a5cc-ede068832de1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:14.799443 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.799423 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.857911 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.857877 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldff8\" (UniqueName: \"kubernetes.io/projected/0b1c6200-5dd2-437c-b3b3-18b919ef7496-kube-api-access-ldff8\") pod \"maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:14.893958 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.893872 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp"] Oct 30 22:04:14.897438 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897403 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-kube-api-access-lshwd\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-kube-api-access-lshwd\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.897653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897510 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-podinfo\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.897653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897567 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd96f278-3477-457e-9a09-56fff149ba61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd96f278-3477-457e-9a09-56fff149ba61-istiod-ca-cert\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.897653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897594 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-envoy\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.897653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897625 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-certs\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.897878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897654 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-kube-api-access-6hn76\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-kube-api-access-6hn76\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.897878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897721 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-socket\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.897878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897756 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-envoy\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.897878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897790 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9935d58c-f632-459c-b66a-1f3843ac59b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9935d58c-f632-459c-b66a-1f3843ac59b3-istiod-ca-cert\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.897878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897826 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-token\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-token\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.897878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897862 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd96f278-3477-457e-9a09-56fff149ba61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd96f278-3477-457e-9a09-56fff149ba61-istio-podinfo\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.898185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897888 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-certs\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.898185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897933 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-istio-token\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-istio-token\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.898185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.897965 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-data\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.898185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898017 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-credential-socket\") pod \"9935d58c-f632-459c-b66a-1f3843ac59b3\" (UID: \"9935d58c-f632-459c-b66a-1f3843ac59b3\") " Oct 30 22:04:14.898185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898053 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-credential-socket\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.898185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898083 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-certs" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.898185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898102 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-socket\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.898185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898151 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-data\") pod \"dd96f278-3477-457e-9a09-56fff149ba61\" (UID: \"dd96f278-3477-457e-9a09-56fff149ba61\") " Oct 30 22:04:14.898626 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898307 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-socket" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.898626 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898511 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-certs" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898849 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898877 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898893 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898911 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xcqp\" (UniqueName: \"kubernetes.io/projected/90bb6a42-f6f9-47be-a5cc-ede068832de1-kube-api-access-5xcqp\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898928 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bb6a42-f6f9-47be-a5cc-ede068832de1-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898942 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898956 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.898089 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd96f278-3477-457e-9a09-56fff149ba61-istiod-ca-cert" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.899178 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-data" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.899323 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7"] Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.899383 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-credential-socket" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.899397 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-data" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.899620 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-socket" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.899697 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-credential-socket" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.899757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.899104 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9935d58c-f632-459c-b66a-1f3843ac59b3-istiod-ca-cert" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:14.900944 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.900891 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd96f278-3477-457e-9a09-56fff149ba61-istio-podinfo" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:14.901068 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.901047 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-envoy" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.902329 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.902303 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-istio-token" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.902938 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.902910 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-kube-api-access-6hn76" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "kube-api-access-6hn76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.903076 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.902983 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-token" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.903445 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.903407 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-kube-api-access-lshwd" pod "dd96f278-3477-457e-9a09-56fff149ba61" (UID: "dd96f278-3477-457e-9a09-56fff149ba61"). InnerVolumeSpecName "kube-api-access-lshwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.904105 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.903944 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-envoy" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.904903 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.904878 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-podinfo" pod "9935d58c-f632-459c-b66a-1f3843ac59b3" (UID: "9935d58c-f632-459c-b66a-1f3843ac59b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:14.935900 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:14.935870 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003075 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003101 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd96f278-3477-457e-9a09-56fff149ba61-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003112 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003121 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hn76\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-kube-api-access-6hn76\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003132 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003141 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003149 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9935d58c-f632-459c-b66a-1f3843ac59b3-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003158 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003168 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd96f278-3477-457e-9a09-56fff149ba61-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003175 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003183 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003192 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9935d58c-f632-459c-b66a-1f3843ac59b3-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003200 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003208 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd96f278-3477-457e-9a09-56fff149ba61-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.003698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.003216 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lshwd\" (UniqueName: \"kubernetes.io/projected/dd96f278-3477-457e-9a09-56fff149ba61-kube-api-access-lshwd\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.330957 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.330919 4526 generic.go:356] "Generic (PLEG): container finished" podID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerID="b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8" exitCode=0 Oct 30 22:04:15.331181 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.330980 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" Oct 30 22:04:15.331181 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.330997 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" event={"ID":"90bb6a42-f6f9-47be-a5cc-ede068832de1","Type":"ContainerDied","Data":"b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8"} Oct 30 22:04:15.331181 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.331059 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c" event={"ID":"90bb6a42-f6f9-47be-a5cc-ede068832de1","Type":"ContainerDied","Data":"e9e12b72343203dd93ec0f5c1b9b13352f65585293bf85fb56faeb900e70dabc"} Oct 30 22:04:15.331181 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.331082 4526 scope.go:122] "RemoveContainer" containerID="b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8" Oct 30 22:04:15.332609 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.332585 4526 generic.go:356] "Generic (PLEG): container finished" podID="dd96f278-3477-457e-9a09-56fff149ba61" containerID="f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56" exitCode=0 Oct 30 22:04:15.332775 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.332638 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" Oct 30 22:04:15.332775 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.332681 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" event={"ID":"dd96f278-3477-457e-9a09-56fff149ba61","Type":"ContainerDied","Data":"f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56"} Oct 30 22:04:15.332775 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.332733 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h" event={"ID":"dd96f278-3477-457e-9a09-56fff149ba61","Type":"ContainerDied","Data":"416f846e2ce191d78559c308adf8e3bb4980a2c4f891bba05abc507ed658297e"} Oct 30 22:04:15.334673 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.334649 4526 generic.go:356] "Generic (PLEG): container finished" podID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerID="c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318" exitCode=0 Oct 30 22:04:15.334841 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.334699 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" Oct 30 22:04:15.334841 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.334704 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" event={"ID":"9935d58c-f632-459c-b66a-1f3843ac59b3","Type":"ContainerDied","Data":"c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318"} Oct 30 22:04:15.334841 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.334811 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt" event={"ID":"9935d58c-f632-459c-b66a-1f3843ac59b3","Type":"ContainerDied","Data":"911a9e8489ab1315667999b196ff1d4168cb5adc4736f8330252fa2277eaae1d"} Oct 30 22:04:15.341644 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.341625 4526 scope.go:122] "RemoveContainer" containerID="9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d" Oct 30 22:04:15.345248 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.345231 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg"] Oct 30 22:04:15.348377 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:04:15.348355 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1c6200_5dd2_437c_b3b3_18b919ef7496.slice/crio-1024d9e96e6c508704f63d03c89730a6563ce8c6d1381de1be31e853d0e95400 WatchSource:0}: Error finding container 1024d9e96e6c508704f63d03c89730a6563ce8c6d1381de1be31e853d0e95400: Status 404 returned error can't find the container with id 1024d9e96e6c508704f63d03c89730a6563ce8c6d1381de1be31e853d0e95400 Oct 30 22:04:15.353948 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.353929 4526 scope.go:122] "RemoveContainer" containerID="b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8" Oct 30 22:04:15.354254 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:15.354190 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8\": container with ID starting with b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8 not found: ID does not exist" containerID="b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8" Oct 30 22:04:15.354254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.354225 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8"} err="failed to get container status \"b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8\": rpc error: code = NotFound desc = could not find container \"b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8\": container with ID starting with b4d7b6cc9e3c2f78bbd8b2c993009afb03e74b347d5f757f6f54ffc6cb1e5eb8 not found: ID does not exist" Oct 30 22:04:15.354254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.354252 4526 scope.go:122] "RemoveContainer" containerID="9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d" Oct 30 22:04:15.354566 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:15.354541 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d\": container with ID starting with 9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d not found: ID does not exist" containerID="9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d" Oct 30 22:04:15.354650 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.354575 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d"} err="failed to get container status \"9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d\": rpc error: code = NotFound desc = could not find container \"9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d\": container with ID starting with 9214643caf5ac7856472b9575a386ba19ab77e13c9af5afb0189231c77d91a5d not found: ID does not exist" Oct 30 22:04:15.354650 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.354595 4526 scope.go:122] "RemoveContainer" containerID="f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56" Oct 30 22:04:15.369686 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.369665 4526 scope.go:122] "RemoveContainer" containerID="f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56" Oct 30 22:04:15.370050 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:15.369980 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56\": container with ID starting with f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56 not found: ID does not exist" containerID="f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56" Oct 30 22:04:15.370050 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.370006 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56"} err="failed to get container status \"f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56\": rpc error: code = NotFound desc = could not find container \"f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56\": container with ID starting with f49f3f19274071ce582d9bd86ba42dd5ae63514e6bd6fd901ec9da54c2272a56 not found: ID does not exist" Oct 30 22:04:15.370050 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.370024 4526 scope.go:122] "RemoveContainer" containerID="c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318" Oct 30 22:04:15.378574 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.378556 4526 scope.go:122] "RemoveContainer" containerID="c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318" Oct 30 22:04:15.378834 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:15.378814 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318\": container with ID starting with c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318 not found: ID does not exist" containerID="c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318" Oct 30 22:04:15.378891 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.378842 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318"} err="failed to get container status \"c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318\": rpc error: code = NotFound desc = could not find container \"c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318\": container with ID starting with c6adae1be09091ea4d84271f61199deb4ab2c4f3d462496da85a249430d80318 not found: ID does not exist" Oct 30 22:04:15.469824 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.469791 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt"] Oct 30 22:04:15.473772 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.473742 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-h8gdt"] Oct 30 22:04:15.587815 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.587724 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c"] Oct 30 22:04:15.591428 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.591401 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-99n7c"] Oct 30 22:04:15.702042 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.701996 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h"] Oct 30 22:04:15.704388 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.704364 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-dfv5h"] Oct 30 22:04:15.984920 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.984839 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" path="/var/lib/kubelet/pods/90bb6a42-f6f9-47be-a5cc-ede068832de1/volumes" Oct 30 22:04:15.985298 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.985284 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" path="/var/lib/kubelet/pods/9935d58c-f632-459c-b66a-1f3843ac59b3/volumes" Oct 30 22:04:15.985707 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:15.985695 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd96f278-3477-457e-9a09-56fff149ba61" path="/var/lib/kubelet/pods/dd96f278-3477-457e-9a09-56fff149ba61/volumes" Oct 30 22:04:16.045829 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:16.045790 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6"] Oct 30 22:04:16.047856 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:16.047829 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d102b1b7-mmfb6"] Oct 30 22:04:16.342780 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:16.342748 4526 generic.go:356] "Generic (PLEG): container finished" podID="0b1c6200-5dd2-437c-b3b3-18b919ef7496" containerID="93460357cc58901f1f36e5900640c4e0b33cbb8a3e440131ce58a3e5701448a6" exitCode=0 Oct 30 22:04:16.342979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:16.342827 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" event={"ID":"0b1c6200-5dd2-437c-b3b3-18b919ef7496","Type":"ContainerDied","Data":"93460357cc58901f1f36e5900640c4e0b33cbb8a3e440131ce58a3e5701448a6"} Oct 30 22:04:16.342979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:16.342876 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" event={"ID":"0b1c6200-5dd2-437c-b3b3-18b919ef7496","Type":"ContainerStarted","Data":"1024d9e96e6c508704f63d03c89730a6563ce8c6d1381de1be31e853d0e95400"} Oct 30 22:04:17.052545 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.052481 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp"] Oct 30 22:04:17.102130 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.102096 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3661db54-p9dwp"] Oct 30 22:04:17.499641 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.499615 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:17.631861 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.631781 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-openshift-ca\") pod \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " Oct 30 22:04:17.631861 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.631821 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1c6200-5dd2-437c-b3b3-18b919ef7496-kube-api-access-ldff8\" (UniqueName: \"kubernetes.io/projected/0b1c6200-5dd2-437c-b3b3-18b919ef7496-kube-api-access-ldff8\") pod \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " Oct 30 22:04:17.631861 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.631849 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-xks-ca\") pod \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\" (UID: \"0b1c6200-5dd2-437c-b3b3-18b919ef7496\") " Oct 30 22:04:17.632256 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.632223 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-openshift-ca" pod "0b1c6200-5dd2-437c-b3b3-18b919ef7496" (UID: "0b1c6200-5dd2-437c-b3b3-18b919ef7496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:17.634163 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.634135 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1c6200-5dd2-437c-b3b3-18b919ef7496-kube-api-access-ldff8" pod "0b1c6200-5dd2-437c-b3b3-18b919ef7496" (UID: "0b1c6200-5dd2-437c-b3b3-18b919ef7496"). InnerVolumeSpecName "kube-api-access-ldff8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:17.634263 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.634184 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-xks-ca" pod "0b1c6200-5dd2-437c-b3b3-18b919ef7496" (UID: "0b1c6200-5dd2-437c-b3b3-18b919ef7496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:17.733234 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.733193 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldff8\" (UniqueName: \"kubernetes.io/projected/0b1c6200-5dd2-437c-b3b3-18b919ef7496-kube-api-access-ldff8\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.733234 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.733227 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.733234 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.733238 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b1c6200-5dd2-437c-b3b3-18b919ef7496-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:17.983938 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.983859 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a33d5c-6993-41bc-a39f-66bd97452b18" path="/var/lib/kubelet/pods/64a33d5c-6993-41bc-a39f-66bd97452b18/volumes" Oct 30 22:04:17.984277 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:17.984264 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed76201b-10ad-4151-a8b0-ba70936b4480" path="/var/lib/kubelet/pods/ed76201b-10ad-4151-a8b0-ba70936b4480/volumes" Oct 30 22:04:18.357007 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:18.356979 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" Oct 30 22:04:18.357007 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:18.356998 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg" event={"ID":"0b1c6200-5dd2-437c-b3b3-18b919ef7496","Type":"ContainerDied","Data":"1024d9e96e6c508704f63d03c89730a6563ce8c6d1381de1be31e853d0e95400"} Oct 30 22:04:18.357608 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:18.357040 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1024d9e96e6c508704f63d03c89730a6563ce8c6d1381de1be31e853d0e95400" Oct 30 22:04:19.065572 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:19.065534 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m"] Oct 30 22:04:19.114105 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:19.114070 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj"] Oct 30 22:04:21.935004 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:21.934966 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9"] Oct 30 22:04:22.038676 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:22.038638 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984"] Oct 30 22:04:25.199055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199017 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7"] Oct 30 22:04:25.199448 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199421 4526 cpu_manager.go:414] "containerMap: removing container" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" Oct 30 22:04:25.199448 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199432 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" Oct 30 22:04:25.199570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199451 4526 cpu_manager.go:414] "containerMap: removing container" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" Oct 30 22:04:25.199570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199456 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" Oct 30 22:04:25.199570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199471 4526 cpu_manager.go:414] "containerMap: removing container" podUID="0b1c6200-5dd2-437c-b3b3-18b919ef7496" containerName="revoke-keys" Oct 30 22:04:25.199570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199476 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1c6200-5dd2-437c-b3b3-18b919ef7496" containerName="revoke-keys" Oct 30 22:04:25.199570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199504 4526 cpu_manager.go:414] "containerMap: removing container" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" Oct 30 22:04:25.199570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199511 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" Oct 30 22:04:25.199570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199527 4526 cpu_manager.go:414] "containerMap: removing container" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" Oct 30 22:04:25.199570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199532 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" Oct 30 22:04:25.199820 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199596 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" Oct 30 22:04:25.199820 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199605 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="9935d58c-f632-459c-b66a-1f3843ac59b3" containerName="istio-proxy" Oct 30 22:04:25.199820 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199615 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd96f278-3477-457e-9a09-56fff149ba61" containerName="istio-proxy" Oct 30 22:04:25.199820 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.199622 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1c6200-5dd2-437c-b3b3-18b919ef7496" containerName="revoke-keys" Oct 30 22:04:25.202545 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.202528 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.210332 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.210283 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw"] Oct 30 22:04:25.212751 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.212379 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bb6a42-f6f9-47be-a5cc-ede068832de1" containerName="istio-proxy" Oct 30 22:04:25.216609 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.216591 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.254377 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.254339 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:25.254570 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.254517 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:25.254654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.254624 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:25.295207 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.295176 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:04:25.319893 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.319858 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.320051 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.319904 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.320051 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.320012 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmww\" (UniqueName: \"kubernetes.io/projected/f90db392-1a2d-4f24-ab28-6ca7b0274107-kube-api-access-7vmww\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.320051 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.320031 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.320202 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.320140 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmcc\" (UniqueName: \"kubernetes.io/projected/72e29896-28f5-41e1-9337-e51dc19299b4-kube-api-access-nzmcc\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.320202 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.320180 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.332006 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.331977 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw"] Oct 30 22:04:25.421427 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.421390 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmww\" (UniqueName: \"kubernetes.io/projected/f90db392-1a2d-4f24-ab28-6ca7b0274107-kube-api-access-7vmww\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.421671 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.421441 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.421671 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.421568 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmcc\" (UniqueName: \"kubernetes.io/projected/72e29896-28f5-41e1-9337-e51dc19299b4-kube-api-access-nzmcc\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.421671 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.421632 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.422151 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.422116 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.422272 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.422192 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.422370 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.422336 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.422965 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.422938 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.425046 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.425023 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.425046 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.425045 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7"] Oct 30 22:04:25.425360 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.425328 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.482104 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.482028 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmcc\" (UniqueName: \"kubernetes.io/projected/72e29896-28f5-41e1-9337-e51dc19299b4-kube-api-access-nzmcc\") pod \"maas-api-revoke-sub-0ee99f3f-s9jbw\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.482258 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.482123 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmww\" (UniqueName: \"kubernetes.io/projected/f90db392-1a2d-4f24-ab28-6ca7b0274107-kube-api-access-7vmww\") pod \"maas-api-revoke-sub-9b1c9f9c-b5qj7\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.523548 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.523513 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:25.531685 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.531655 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:25.902577 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.902552 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7"] Oct 30 22:04:25.904693 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:04:25.904658 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90db392_1a2d_4f24_ab28_6ca7b0274107.slice/crio-45605b475eb19eac492d862aabc5145913d8d7b3b9c9a669670fc165ba68fa40 WatchSource:0}: Error finding container 45605b475eb19eac492d862aabc5145913d8d7b3b9c9a669670fc165ba68fa40: Status 404 returned error can't find the container with id 45605b475eb19eac492d862aabc5145913d8d7b3b9c9a669670fc165ba68fa40 Oct 30 22:04:25.953439 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:25.953407 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw"] Oct 30 22:04:25.955343 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:04:25.955316 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e29896_28f5_41e1_9337_e51dc19299b4.slice/crio-f00efe83b841c731aabb83dc96c06f6d75c4723c012825bafc43b300abb89fb2 WatchSource:0}: Error finding container f00efe83b841c731aabb83dc96c06f6d75c4723c012825bafc43b300abb89fb2: Status 404 returned error can't find the container with id f00efe83b841c731aabb83dc96c06f6d75c4723c012825bafc43b300abb89fb2 Oct 30 22:04:26.116227 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:26.116201 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc"] Oct 30 22:04:26.239086 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:26.239056 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq"] Oct 30 22:04:26.406808 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:26.406768 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" event={"ID":"72e29896-28f5-41e1-9337-e51dc19299b4","Type":"ContainerStarted","Data":"a02a67e9cf8209b4beb9d04b5feaf1df59279f4480814398647694a5fd8a82db"} Oct 30 22:04:26.407004 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:26.406840 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" event={"ID":"72e29896-28f5-41e1-9337-e51dc19299b4","Type":"ContainerStarted","Data":"f00efe83b841c731aabb83dc96c06f6d75c4723c012825bafc43b300abb89fb2"} Oct 30 22:04:26.408352 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:26.408325 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" event={"ID":"f90db392-1a2d-4f24-ab28-6ca7b0274107","Type":"ContainerStarted","Data":"68b2c95e0971d2cac8e256cd256a579fb4b27c7552579673879c03820661087a"} Oct 30 22:04:26.408528 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:26.408366 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" event={"ID":"f90db392-1a2d-4f24-ab28-6ca7b0274107","Type":"ContainerStarted","Data":"45605b475eb19eac492d862aabc5145913d8d7b3b9c9a669670fc165ba68fa40"} Oct 30 22:04:26.524785 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:26.524668 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" podStartSLOduration=1.524653667 podStartE2EDuration="1.524653667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:26.521646052 +0000 UTC m=+2856.655019250" watchObservedRunningTime="2026-10-30 22:04:26.524653667 +0000 UTC m=+2856.658026865" Oct 30 22:04:26.935434 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:26.935363 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" podUID="79acc862-069d-4c9e-9759-535067b6e745" containerName="payload-processing" containerID="cri-o://398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6" gracePeriod=25 Oct 30 22:04:27.039652 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:27.039587 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" podUID="6b9e0bc4-bf67-413f-a580-974404f77720" containerName="payload-pre-processing" containerID="cri-o://6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e" gracePeriod=25 Oct 30 22:04:27.414818 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:27.414785 4526 generic.go:356] "Generic (PLEG): container finished" podID="72e29896-28f5-41e1-9337-e51dc19299b4" containerID="a02a67e9cf8209b4beb9d04b5feaf1df59279f4480814398647694a5fd8a82db" exitCode=0 Oct 30 22:04:27.415263 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:27.414879 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" event={"ID":"72e29896-28f5-41e1-9337-e51dc19299b4","Type":"ContainerDied","Data":"a02a67e9cf8209b4beb9d04b5feaf1df59279f4480814398647694a5fd8a82db"} Oct 30 22:04:27.416538 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:27.416519 4526 generic.go:356] "Generic (PLEG): container finished" podID="f90db392-1a2d-4f24-ab28-6ca7b0274107" containerID="68b2c95e0971d2cac8e256cd256a579fb4b27c7552579673879c03820661087a" exitCode=0 Oct 30 22:04:27.416595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:27.416573 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" event={"ID":"f90db392-1a2d-4f24-ab28-6ca7b0274107","Type":"ContainerDied","Data":"68b2c95e0971d2cac8e256cd256a579fb4b27c7552579673879c03820661087a"} Oct 30 22:04:27.541937 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:27.541882 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" podStartSLOduration=2.5418661030000003 podStartE2EDuration="2.541866103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:26.637719806 +0000 UTC m=+2856.771093004" watchObservedRunningTime="2026-10-30 22:04:27.541866103 +0000 UTC m=+2857.675239301" Oct 30 22:04:28.571439 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.571416 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:28.593613 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.593586 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:28.651462 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.651423 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-xks-ca\") pod \"72e29896-28f5-41e1-9337-e51dc19299b4\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " Oct 30 22:04:28.651670 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.651472 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90db392-1a2d-4f24-ab28-6ca7b0274107-kube-api-access-7vmww\" (UniqueName: \"kubernetes.io/projected/f90db392-1a2d-4f24-ab28-6ca7b0274107-kube-api-access-7vmww\") pod \"f90db392-1a2d-4f24-ab28-6ca7b0274107\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " Oct 30 22:04:28.651670 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.651623 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e29896-28f5-41e1-9337-e51dc19299b4-kube-api-access-nzmcc\" (UniqueName: \"kubernetes.io/projected/72e29896-28f5-41e1-9337-e51dc19299b4-kube-api-access-nzmcc\") pod \"72e29896-28f5-41e1-9337-e51dc19299b4\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " Oct 30 22:04:28.651766 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.651694 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-openshift-ca\") pod \"f90db392-1a2d-4f24-ab28-6ca7b0274107\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " Oct 30 22:04:28.651766 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.651746 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-openshift-ca\") pod \"72e29896-28f5-41e1-9337-e51dc19299b4\" (UID: \"72e29896-28f5-41e1-9337-e51dc19299b4\") " Oct 30 22:04:28.651847 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.651789 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-xks-ca\") pod \"f90db392-1a2d-4f24-ab28-6ca7b0274107\" (UID: \"f90db392-1a2d-4f24-ab28-6ca7b0274107\") " Oct 30 22:04:28.652152 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.652079 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-openshift-ca" pod "f90db392-1a2d-4f24-ab28-6ca7b0274107" (UID: "f90db392-1a2d-4f24-ab28-6ca7b0274107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:28.652152 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.652130 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-openshift-ca" pod "72e29896-28f5-41e1-9337-e51dc19299b4" (UID: "72e29896-28f5-41e1-9337-e51dc19299b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:28.652343 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.652183 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:28.653915 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.653883 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-xks-ca" pod "72e29896-28f5-41e1-9337-e51dc19299b4" (UID: "72e29896-28f5-41e1-9337-e51dc19299b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:28.654027 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.653983 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-xks-ca" pod "f90db392-1a2d-4f24-ab28-6ca7b0274107" (UID: "f90db392-1a2d-4f24-ab28-6ca7b0274107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:28.654391 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.654369 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e29896-28f5-41e1-9337-e51dc19299b4-kube-api-access-nzmcc" pod "72e29896-28f5-41e1-9337-e51dc19299b4" (UID: "72e29896-28f5-41e1-9337-e51dc19299b4"). InnerVolumeSpecName "kube-api-access-nzmcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:28.654516 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.654397 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90db392-1a2d-4f24-ab28-6ca7b0274107-kube-api-access-7vmww" pod "f90db392-1a2d-4f24-ab28-6ca7b0274107" (UID: "f90db392-1a2d-4f24-ab28-6ca7b0274107"). InnerVolumeSpecName "kube-api-access-7vmww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:28.753354 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.753264 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f90db392-1a2d-4f24-ab28-6ca7b0274107-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:28.753354 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.753296 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:28.753354 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.753307 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vmww\" (UniqueName: \"kubernetes.io/projected/f90db392-1a2d-4f24-ab28-6ca7b0274107-kube-api-access-7vmww\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:28.753354 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.753316 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzmcc\" (UniqueName: \"kubernetes.io/projected/72e29896-28f5-41e1-9337-e51dc19299b4-kube-api-access-nzmcc\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:28.753354 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:28.753327 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72e29896-28f5-41e1-9337-e51dc19299b4-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:29.026690 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.026596 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" podUID="79acc862-069d-4c9e-9759-535067b6e745" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:29.026690 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.026609 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" podUID="6b9e0bc4-bf67-413f-a580-974404f77720" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:29.428986 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.428953 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" Oct 30 22:04:29.429161 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.428989 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw" event={"ID":"72e29896-28f5-41e1-9337-e51dc19299b4","Type":"ContainerDied","Data":"f00efe83b841c731aabb83dc96c06f6d75c4723c012825bafc43b300abb89fb2"} Oct 30 22:04:29.429161 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.429046 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f00efe83b841c731aabb83dc96c06f6d75c4723c012825bafc43b300abb89fb2" Oct 30 22:04:29.430266 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.430244 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" Oct 30 22:04:29.430266 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.430254 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7" event={"ID":"f90db392-1a2d-4f24-ab28-6ca7b0274107","Type":"ContainerDied","Data":"45605b475eb19eac492d862aabc5145913d8d7b3b9c9a669670fc165ba68fa40"} Oct 30 22:04:29.430430 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.430298 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45605b475eb19eac492d862aabc5145913d8d7b3b9c9a669670fc165ba68fa40" Oct 30 22:04:29.917630 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.917588 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" containerID="cri-o://42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da" gracePeriod=45 Oct 30 22:04:29.920767 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:29.920708 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" containerID="cri-o://d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50" gracePeriod=45 Oct 30 22:04:30.062441 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.062394 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": dial tcp 10.133.0.88:8000: connect: connection refused" Oct 30 22:04:30.177551 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.177464 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 22:04:30.193048 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.193022 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 22:04:30.194279 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194256 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl"] Oct 30 22:04:30.194728 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194711 4526 cpu_manager.go:414] "containerMap: removing container" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" Oct 30 22:04:30.194728 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194728 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" Oct 30 22:04:30.194728 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194740 4526 cpu_manager.go:414] "containerMap: removing container" podUID="72e29896-28f5-41e1-9337-e51dc19299b4" containerName="revoke-keys" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194746 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e29896-28f5-41e1-9337-e51dc19299b4" containerName="revoke-keys" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194769 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194776 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194783 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f90db392-1a2d-4f24-ab28-6ca7b0274107" containerName="revoke-keys" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194787 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90db392-1a2d-4f24-ab28-6ca7b0274107" containerName="revoke-keys" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194861 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerName="main" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194869 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90db392-1a2d-4f24-ab28-6ca7b0274107" containerName="revoke-keys" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194881 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerName="main" Oct 30 22:04:30.194895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.194888 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e29896-28f5-41e1-9337-e51dc19299b4" containerName="revoke-keys" Oct 30 22:04:30.197826 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.197811 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.249995 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.249963 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln"] Oct 30 22:04:30.250685 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.250656 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:04:30.251438 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.251416 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:30.253761 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.253738 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.268762 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.268729 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tls-certs\") pod \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " Oct 30 22:04:30.268958 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.268779 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83fd981-a59c-4a55-b675-d19b4bab1d5e-kube-api-access-9j5rt\" (UniqueName: \"kubernetes.io/projected/a83fd981-a59c-4a55-b675-d19b4bab1d5e-kube-api-access-9j5rt\") pod \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " Oct 30 22:04:30.268958 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.268805 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-model-cache\") pod \"9e930fb0-4c95-4e49-aba1-2db4356980df\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " Oct 30 22:04:30.268958 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.268861 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-tmp-dir\") pod \"9e930fb0-4c95-4e49-aba1-2db4356980df\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " Oct 30 22:04:30.268958 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.268904 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e930fb0-4c95-4e49-aba1-2db4356980df-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e930fb0-4c95-4e49-aba1-2db4356980df-tls-certs\") pod \"9e930fb0-4c95-4e49-aba1-2db4356980df\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " Oct 30 22:04:30.268958 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.268956 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e930fb0-4c95-4e49-aba1-2db4356980df-kube-api-access-hp6ls\" (UniqueName: \"kubernetes.io/projected/9e930fb0-4c95-4e49-aba1-2db4356980df-kube-api-access-hp6ls\") pod \"9e930fb0-4c95-4e49-aba1-2db4356980df\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " Oct 30 22:04:30.269257 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.268999 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-dshm\") pod \"9e930fb0-4c95-4e49-aba1-2db4356980df\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " Oct 30 22:04:30.269257 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269057 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tmp-dir\") pod \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " Oct 30 22:04:30.269257 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269093 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-home\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-home\") pod \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " Oct 30 22:04:30.269257 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269141 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-tmp-dir" pod "9e930fb0-4c95-4e49-aba1-2db4356980df" (UID: "9e930fb0-4c95-4e49-aba1-2db4356980df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.269257 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269151 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-dshm\") pod \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " Oct 30 22:04:30.269257 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269221 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-home\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-home\") pod \"9e930fb0-4c95-4e49-aba1-2db4356980df\" (UID: \"9e930fb0-4c95-4e49-aba1-2db4356980df\") " Oct 30 22:04:30.269634 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269259 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-model-cache\") pod \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\" (UID: \"a83fd981-a59c-4a55-b675-d19b4bab1d5e\") " Oct 30 22:04:30.269634 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269424 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.269634 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269433 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-model-cache" pod "9e930fb0-4c95-4e49-aba1-2db4356980df" (UID: "9e930fb0-4c95-4e49-aba1-2db4356980df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.269634 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269479 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.269634 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269570 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.269928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269637 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhwx\" (UniqueName: \"kubernetes.io/projected/b1df5bca-fc84-4bec-bd46-cd7b804234de-kube-api-access-zmhwx\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.269928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269670 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tmp-dir" pod "a83fd981-a59c-4a55-b675-d19b4bab1d5e" (UID: "a83fd981-a59c-4a55-b675-d19b4bab1d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.269928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269868 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-home" pod "a83fd981-a59c-4a55-b675-d19b4bab1d5e" (UID: "a83fd981-a59c-4a55-b675-d19b4bab1d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.269928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.269872 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-home" pod "9e930fb0-4c95-4e49-aba1-2db4356980df" (UID: "9e930fb0-4c95-4e49-aba1-2db4356980df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.270167 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270047 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-model-cache" pod "a83fd981-a59c-4a55-b675-d19b4bab1d5e" (UID: "a83fd981-a59c-4a55-b675-d19b4bab1d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.270227 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270196 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnkc2\" (UniqueName: \"kubernetes.io/projected/57944f20-befb-4a33-8288-1aa2b87024de-kube-api-access-nnkc2\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270287 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270373 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270391 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270404 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270416 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270429 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.270443 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.272242 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-dshm" pod "a83fd981-a59c-4a55-b675-d19b4bab1d5e" (UID: "a83fd981-a59c-4a55-b675-d19b4bab1d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.272358 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83fd981-a59c-4a55-b675-d19b4bab1d5e-kube-api-access-9j5rt" pod "a83fd981-a59c-4a55-b675-d19b4bab1d5e" (UID: "a83fd981-a59c-4a55-b675-d19b4bab1d5e"). InnerVolumeSpecName "kube-api-access-9j5rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.272458 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-dshm" pod "9e930fb0-4c95-4e49-aba1-2db4356980df" (UID: "9e930fb0-4c95-4e49-aba1-2db4356980df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.272475 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tls-certs" pod "a83fd981-a59c-4a55-b675-d19b4bab1d5e" (UID: "a83fd981-a59c-4a55-b675-d19b4bab1d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.273611 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e930fb0-4c95-4e49-aba1-2db4356980df-tls-certs" pod "9e930fb0-4c95-4e49-aba1-2db4356980df" (UID: "9e930fb0-4c95-4e49-aba1-2db4356980df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:30.274259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.273741 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e930fb0-4c95-4e49-aba1-2db4356980df-kube-api-access-hp6ls" pod "9e930fb0-4c95-4e49-aba1-2db4356980df" (UID: "9e930fb0-4c95-4e49-aba1-2db4356980df"). InnerVolumeSpecName "kube-api-access-hp6ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:30.278468 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.278445 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:30.278468 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.278456 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:30.280782 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.280758 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl"] Oct 30 22:04:30.369264 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.369223 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln"] Oct 30 22:04:30.371128 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371095 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnkc2\" (UniqueName: \"kubernetes.io/projected/57944f20-befb-4a33-8288-1aa2b87024de-kube-api-access-nnkc2\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.371309 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371199 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.371309 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371253 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.371309 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371293 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.371466 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371345 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.371466 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371386 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhwx\" (UniqueName: \"kubernetes.io/projected/b1df5bca-fc84-4bec-bd46-cd7b804234de-kube-api-access-zmhwx\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.371606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371477 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e930fb0-4c95-4e49-aba1-2db4356980df-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.371606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371528 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6ls\" (UniqueName: \"kubernetes.io/projected/9e930fb0-4c95-4e49-aba1-2db4356980df-kube-api-access-hp6ls\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.371606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371544 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e930fb0-4c95-4e49-aba1-2db4356980df-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.371606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371560 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83fd981-a59c-4a55-b675-d19b4bab1d5e-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.371606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371573 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83fd981-a59c-4a55-b675-d19b4bab1d5e-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.371606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.371586 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5rt\" (UniqueName: \"kubernetes.io/projected/a83fd981-a59c-4a55-b675-d19b4bab1d5e-kube-api-access-9j5rt\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.372222 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.372193 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.372465 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.372437 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.375021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.374996 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.375125 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.374996 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.437847 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.437757 4526 generic.go:356] "Generic (PLEG): container finished" podID="9e930fb0-4c95-4e49-aba1-2db4356980df" containerID="42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da" exitCode=0 Oct 30 22:04:30.437847 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.437797 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" Oct 30 22:04:30.438079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.437842 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" event={"ID":"9e930fb0-4c95-4e49-aba1-2db4356980df","Type":"ContainerDied","Data":"42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da"} Oct 30 22:04:30.438079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.437885 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp" event={"ID":"9e930fb0-4c95-4e49-aba1-2db4356980df","Type":"ContainerDied","Data":"b197aa5adbdcb04f3487f0350d9abf7cec628ac7922e0fe85a80a556eb036291"} Oct 30 22:04:30.438079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.437911 4526 scope.go:122] "RemoveContainer" containerID="42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da" Oct 30 22:04:30.439447 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.439427 4526 generic.go:356] "Generic (PLEG): container finished" podID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" containerID="d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50" exitCode=0 Oct 30 22:04:30.439577 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.439538 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" event={"ID":"a83fd981-a59c-4a55-b675-d19b4bab1d5e","Type":"ContainerDied","Data":"d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50"} Oct 30 22:04:30.439577 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.439561 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" event={"ID":"a83fd981-a59c-4a55-b675-d19b4bab1d5e","Type":"ContainerDied","Data":"092aba4c87a0e3fec57d4a3a60eda8ef4be93fcc329b1b2c8a0f2ff7114c7063"} Oct 30 22:04:30.439688 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.439611 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7" Oct 30 22:04:30.441009 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.440978 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhwx\" (UniqueName: \"kubernetes.io/projected/b1df5bca-fc84-4bec-bd46-cd7b804234de-kube-api-access-zmhwx\") pod \"maas-api-revoke-sub-772a8b8a-t48ln\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.445001 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.444977 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnkc2\" (UniqueName: \"kubernetes.io/projected/57944f20-befb-4a33-8288-1aa2b87024de-kube-api-access-nnkc2\") pod \"maas-api-revoke-sub-7a81e38c-xdfjl\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.448654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.448635 4526 scope.go:122] "RemoveContainer" containerID="42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da" Oct 30 22:04:30.448987 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:30.448966 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da\": container with ID starting with 42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da not found: ID does not exist" containerID="42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da" Oct 30 22:04:30.449030 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.448999 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da"} err="failed to get container status \"42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da\": rpc error: code = NotFound desc = could not find container \"42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da\": container with ID starting with 42ec31a64ad849ee00cb62de7b5884730d9f5ab28ca5445556f9bbe99e5f00da not found: ID does not exist" Oct 30 22:04:30.449074 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.449027 4526 scope.go:122] "RemoveContainer" containerID="d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50" Oct 30 22:04:30.458668 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.458645 4526 scope.go:122] "RemoveContainer" containerID="d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50" Oct 30 22:04:30.458951 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:30.458932 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50\": container with ID starting with d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50 not found: ID does not exist" containerID="d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50" Oct 30 22:04:30.459003 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.458959 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50"} err="failed to get container status \"d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50\": rpc error: code = NotFound desc = could not find container \"d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50\": container with ID starting with d3820ec7cfadba24f3bc0761ecc50b1b0ac52ed05de520b4974eb10779088f50 not found: ID does not exist" Oct 30 22:04:30.511212 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.511183 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:30.570751 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.570724 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:30.819370 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.819332 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7"] Oct 30 22:04:30.828413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.828382 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-w4-07fa8e-kserve-7f87ccdfcd-p87f7"] Oct 30 22:04:30.935333 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.935299 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp"] Oct 30 22:04:30.940869 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.940752 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl"] Oct 30 22:04:30.942677 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.942653 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/distinct-2-w4-07fa8e-kserve-667ffb65cc-l9xgp"] Oct 30 22:04:30.943119 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:04:30.943093 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57944f20_befb_4a33_8288_1aa2b87024de.slice/crio-7b02ff380c38fc0a3248b4127dca2aa0e077701ec96b2865e79f5c291eabb240 WatchSource:0}: Error finding container 7b02ff380c38fc0a3248b4127dca2aa0e077701ec96b2865e79f5c291eabb240: Status 404 returned error can't find the container with id 7b02ff380c38fc0a3248b4127dca2aa0e077701ec96b2865e79f5c291eabb240 Oct 30 22:04:30.992953 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:30.992924 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln"] Oct 30 22:04:31.000113 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:04:31.000088 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1df5bca_fc84_4bec_bd46_cd7b804234de.slice/crio-44f0b125c0a92f84de817ff273321fd764840059fd7d4955dfe9fe92291d0428 WatchSource:0}: Error finding container 44f0b125c0a92f84de817ff273321fd764840059fd7d4955dfe9fe92291d0428: Status 404 returned error can't find the container with id 44f0b125c0a92f84de817ff273321fd764840059fd7d4955dfe9fe92291d0428 Oct 30 22:04:31.117294 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.117234 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" podUID="af111955-96d6-46e2-8103-2e4f51b8a622" containerName="payload-processing" containerID="cri-o://d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da" gracePeriod=25 Oct 30 22:04:31.239865 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.239813 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" podUID="06dfcdda-bbd5-4ebb-9829-f6c23e06982f" containerName="payload-pre-processing" containerID="cri-o://59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0" gracePeriod=25 Oct 30 22:04:31.364259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.364234 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 22:04:31.381624 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.381588 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af111955-96d6-46e2-8103-2e4f51b8a622-kube-api-access-l8x59\" (UniqueName: \"kubernetes.io/projected/af111955-96d6-46e2-8103-2e4f51b8a622-kube-api-access-l8x59\") pod \"af111955-96d6-46e2-8103-2e4f51b8a622\" (UID: \"af111955-96d6-46e2-8103-2e4f51b8a622\") " Oct 30 22:04:31.381772 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.381698 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af111955-96d6-46e2-8103-2e4f51b8a622-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af111955-96d6-46e2-8103-2e4f51b8a622-plugins-config-volume\") pod \"af111955-96d6-46e2-8103-2e4f51b8a622\" (UID: \"af111955-96d6-46e2-8103-2e4f51b8a622\") " Oct 30 22:04:31.382082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.382053 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af111955-96d6-46e2-8103-2e4f51b8a622-plugins-config-volume" pod "af111955-96d6-46e2-8103-2e4f51b8a622" (UID: "af111955-96d6-46e2-8103-2e4f51b8a622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:31.383940 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.383921 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af111955-96d6-46e2-8103-2e4f51b8a622-kube-api-access-l8x59" pod "af111955-96d6-46e2-8103-2e4f51b8a622" (UID: "af111955-96d6-46e2-8103-2e4f51b8a622"). InnerVolumeSpecName "kube-api-access-l8x59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:31.446506 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.446454 4526 generic.go:356] "Generic (PLEG): container finished" podID="af111955-96d6-46e2-8103-2e4f51b8a622" containerID="d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da" exitCode=0 Oct 30 22:04:31.446681 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.446530 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" Oct 30 22:04:31.446681 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.446527 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" event={"ID":"af111955-96d6-46e2-8103-2e4f51b8a622","Type":"ContainerDied","Data":"d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da"} Oct 30 22:04:31.446681 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.446664 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc" event={"ID":"af111955-96d6-46e2-8103-2e4f51b8a622","Type":"ContainerDied","Data":"5dc3393b4fc5ef5e11b8c249e9f893f25d4acd728cf25e77f89212ee6031d10b"} Oct 30 22:04:31.446883 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.446685 4526 scope.go:122] "RemoveContainer" containerID="d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da" Oct 30 22:04:31.451341 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.451313 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" event={"ID":"57944f20-befb-4a33-8288-1aa2b87024de","Type":"ContainerStarted","Data":"53c0cff847c29f016fae10a3e780daa921cb4af8011c85f34bad5d740e56f40a"} Oct 30 22:04:31.451465 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.451374 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" event={"ID":"57944f20-befb-4a33-8288-1aa2b87024de","Type":"ContainerStarted","Data":"7b02ff380c38fc0a3248b4127dca2aa0e077701ec96b2865e79f5c291eabb240"} Oct 30 22:04:31.453321 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.453289 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" event={"ID":"b1df5bca-fc84-4bec-bd46-cd7b804234de","Type":"ContainerStarted","Data":"1bb78fb0319957136b4994bdaca9edd2c20ab9528f73e71ca3aad8b6c2bd6e7f"} Oct 30 22:04:31.453441 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.453346 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" event={"ID":"b1df5bca-fc84-4bec-bd46-cd7b804234de","Type":"ContainerStarted","Data":"44f0b125c0a92f84de817ff273321fd764840059fd7d4955dfe9fe92291d0428"} Oct 30 22:04:31.457956 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.457938 4526 scope.go:122] "RemoveContainer" containerID="d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da" Oct 30 22:04:31.458220 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:31.458200 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da\": container with ID starting with d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da not found: ID does not exist" containerID="d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da" Oct 30 22:04:31.458278 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.458227 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da"} err="failed to get container status \"d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da\": rpc error: code = NotFound desc = could not find container \"d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da\": container with ID starting with d783995160f6aa8d40845f29a9c9c41d8832b3b3c00aa80ec7702ef19fc587da not found: ID does not exist" Oct 30 22:04:31.482928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.482839 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8x59\" (UniqueName: \"kubernetes.io/projected/af111955-96d6-46e2-8103-2e4f51b8a622-kube-api-access-l8x59\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:31.482928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.482870 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af111955-96d6-46e2-8103-2e4f51b8a622-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:31.587944 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.587913 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc"] Oct 30 22:04:31.589909 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.589893 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-446ca2-5db4694b8c-2t9sc"] Oct 30 22:04:31.700651 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.700550 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" podStartSLOduration=1.7005330330000001 podStartE2EDuration="1.700533033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:31.698990483 +0000 UTC m=+2861.832363681" watchObservedRunningTime="2026-10-30 22:04:31.700533033 +0000 UTC m=+2861.833906232" Oct 30 22:04:31.813107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.812908 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" podStartSLOduration=1.81289232 podStartE2EDuration="1.81289232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:31.811933942 +0000 UTC m=+2861.945307157" watchObservedRunningTime="2026-10-30 22:04:31.81289232 +0000 UTC m=+2861.946265524" Oct 30 22:04:31.984220 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.984179 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e930fb0-4c95-4e49-aba1-2db4356980df" path="/var/lib/kubelet/pods/9e930fb0-4c95-4e49-aba1-2db4356980df/volumes" Oct 30 22:04:31.984809 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.984790 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83fd981-a59c-4a55-b675-d19b4bab1d5e" path="/var/lib/kubelet/pods/a83fd981-a59c-4a55-b675-d19b4bab1d5e/volumes" Oct 30 22:04:31.985184 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:31.985165 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af111955-96d6-46e2-8103-2e4f51b8a622" path="/var/lib/kubelet/pods/af111955-96d6-46e2-8103-2e4f51b8a622/volumes" Oct 30 22:04:32.460361 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:32.460325 4526 generic.go:356] "Generic (PLEG): container finished" podID="57944f20-befb-4a33-8288-1aa2b87024de" containerID="53c0cff847c29f016fae10a3e780daa921cb4af8011c85f34bad5d740e56f40a" exitCode=0 Oct 30 22:04:32.460592 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:32.460399 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" event={"ID":"57944f20-befb-4a33-8288-1aa2b87024de","Type":"ContainerDied","Data":"53c0cff847c29f016fae10a3e780daa921cb4af8011c85f34bad5d740e56f40a"} Oct 30 22:04:32.469405 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:32.469370 4526 generic.go:356] "Generic (PLEG): container finished" podID="b1df5bca-fc84-4bec-bd46-cd7b804234de" containerID="1bb78fb0319957136b4994bdaca9edd2c20ab9528f73e71ca3aad8b6c2bd6e7f" exitCode=0 Oct 30 22:04:32.469622 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:32.469413 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" event={"ID":"b1df5bca-fc84-4bec-bd46-cd7b804234de","Type":"ContainerDied","Data":"1bb78fb0319957136b4994bdaca9edd2c20ab9528f73e71ca3aad8b6c2bd6e7f"} Oct 30 22:04:33.642593 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.642568 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:33.644632 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.644614 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:33.705239 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705207 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-openshift-ca\") pod \"b1df5bca-fc84-4bec-bd46-cd7b804234de\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " Oct 30 22:04:33.705431 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705266 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-xks-ca\") pod \"b1df5bca-fc84-4bec-bd46-cd7b804234de\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " Oct 30 22:04:33.705431 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705357 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57944f20-befb-4a33-8288-1aa2b87024de-kube-api-access-nnkc2\" (UniqueName: \"kubernetes.io/projected/57944f20-befb-4a33-8288-1aa2b87024de-kube-api-access-nnkc2\") pod \"57944f20-befb-4a33-8288-1aa2b87024de\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " Oct 30 22:04:33.705431 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705392 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-xks-ca\") pod \"57944f20-befb-4a33-8288-1aa2b87024de\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " Oct 30 22:04:33.705622 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705448 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-openshift-ca\") pod \"57944f20-befb-4a33-8288-1aa2b87024de\" (UID: \"57944f20-befb-4a33-8288-1aa2b87024de\") " Oct 30 22:04:33.705622 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705535 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1df5bca-fc84-4bec-bd46-cd7b804234de-kube-api-access-zmhwx\" (UniqueName: \"kubernetes.io/projected/b1df5bca-fc84-4bec-bd46-cd7b804234de-kube-api-access-zmhwx\") pod \"b1df5bca-fc84-4bec-bd46-cd7b804234de\" (UID: \"b1df5bca-fc84-4bec-bd46-cd7b804234de\") " Oct 30 22:04:33.705723 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705646 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-openshift-ca" pod "b1df5bca-fc84-4bec-bd46-cd7b804234de" (UID: "b1df5bca-fc84-4bec-bd46-cd7b804234de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:33.705874 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705844 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-openshift-ca" pod "57944f20-befb-4a33-8288-1aa2b87024de" (UID: "57944f20-befb-4a33-8288-1aa2b87024de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:33.706020 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.705988 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:33.706020 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.706010 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:33.707855 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.707826 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57944f20-befb-4a33-8288-1aa2b87024de-kube-api-access-nnkc2" pod "57944f20-befb-4a33-8288-1aa2b87024de" (UID: "57944f20-befb-4a33-8288-1aa2b87024de"). InnerVolumeSpecName "kube-api-access-nnkc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:33.708196 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.708177 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-xks-ca" pod "b1df5bca-fc84-4bec-bd46-cd7b804234de" (UID: "b1df5bca-fc84-4bec-bd46-cd7b804234de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:33.708397 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.708372 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-xks-ca" pod "57944f20-befb-4a33-8288-1aa2b87024de" (UID: "57944f20-befb-4a33-8288-1aa2b87024de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:33.708507 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.708394 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1df5bca-fc84-4bec-bd46-cd7b804234de-kube-api-access-zmhwx" pod "b1df5bca-fc84-4bec-bd46-cd7b804234de" (UID: "b1df5bca-fc84-4bec-bd46-cd7b804234de"). InnerVolumeSpecName "kube-api-access-zmhwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:33.806539 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.806470 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1df5bca-fc84-4bec-bd46-cd7b804234de-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:33.806539 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.806537 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnkc2\" (UniqueName: \"kubernetes.io/projected/57944f20-befb-4a33-8288-1aa2b87024de-kube-api-access-nnkc2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:33.806539 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.806548 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57944f20-befb-4a33-8288-1aa2b87024de-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:33.806809 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:33.806558 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhwx\" (UniqueName: \"kubernetes.io/projected/b1df5bca-fc84-4bec-bd46-cd7b804234de-kube-api-access-zmhwx\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.072617 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.072530 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" podUID="06dfcdda-bbd5-4ebb-9829-f6c23e06982f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:34.086269 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.086223 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" containerID="cri-o://4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8" gracePeriod=45 Oct 30 22:04:34.134604 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.134560 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" containerID="cri-o://a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651" gracePeriod=45 Oct 30 22:04:34.297895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.297856 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused" Oct 30 22:04:34.378136 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.378109 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 22:04:34.410639 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.410611 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 22:04:34.412767 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.412725 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tls-certs\") pod \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " Oct 30 22:04:34.412865 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.412815 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-model-cache\") pod \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " Oct 30 22:04:34.412927 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.412913 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tmp-dir\") pod \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " Oct 30 22:04:34.412996 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.412981 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-kube-api-access-pr9k6\" (UniqueName: \"kubernetes.io/projected/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-kube-api-access-pr9k6\") pod \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " Oct 30 22:04:34.413051 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.413037 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-dshm\") pod \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " Oct 30 22:04:34.413118 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.413082 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-home\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-home\") pod \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\" (UID: \"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0\") " Oct 30 22:04:34.413118 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.413081 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-model-cache" pod "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" (UID: "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.413228 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.413169 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tmp-dir" pod "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" (UID: "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.413411 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.413387 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-home" pod "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" (UID: "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.413528 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.413447 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.413607 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.413482 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.413664 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.413610 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.415227 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.415203 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-dshm" pod "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" (UID: "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.415320 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.415240 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tls-certs" pod "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" (UID: "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:34.415320 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.415280 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-kube-api-access-pr9k6" pod "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" (UID: "cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0"). InnerVolumeSpecName "kube-api-access-pr9k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:34.484876 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.484841 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" Oct 30 22:04:34.484876 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.484857 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl" event={"ID":"57944f20-befb-4a33-8288-1aa2b87024de","Type":"ContainerDied","Data":"7b02ff380c38fc0a3248b4127dca2aa0e077701ec96b2865e79f5c291eabb240"} Oct 30 22:04:34.485082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.484912 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b02ff380c38fc0a3248b4127dca2aa0e077701ec96b2865e79f5c291eabb240" Oct 30 22:04:34.486378 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.486353 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" Oct 30 22:04:34.486566 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.486347 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln" event={"ID":"b1df5bca-fc84-4bec-bd46-cd7b804234de","Type":"ContainerDied","Data":"44f0b125c0a92f84de817ff273321fd764840059fd7d4955dfe9fe92291d0428"} Oct 30 22:04:34.486566 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.486483 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44f0b125c0a92f84de817ff273321fd764840059fd7d4955dfe9fe92291d0428" Oct 30 22:04:34.488088 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.488067 4526 generic.go:356] "Generic (PLEG): container finished" podID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerID="a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651" exitCode=0 Oct 30 22:04:34.488201 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.488126 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" Oct 30 22:04:34.488201 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.488153 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" event={"ID":"32765e94-ffd5-4160-b1bd-0636b0f5ac86","Type":"ContainerDied","Data":"a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651"} Oct 30 22:04:34.488333 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.488202 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj" event={"ID":"32765e94-ffd5-4160-b1bd-0636b0f5ac86","Type":"ContainerDied","Data":"99b6e78b33657fcd7ed82c2ef7b132025968e718073bbf3e97436f3142fdd7a9"} Oct 30 22:04:34.488333 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.488221 4526 scope.go:122] "RemoveContainer" containerID="a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651" Oct 30 22:04:34.490314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.490050 4526 generic.go:356] "Generic (PLEG): container finished" podID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerID="4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8" exitCode=0 Oct 30 22:04:34.490314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.490078 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" Oct 30 22:04:34.490314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.490127 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" event={"ID":"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0","Type":"ContainerDied","Data":"4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8"} Oct 30 22:04:34.490314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.490166 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" event={"ID":"cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0","Type":"ContainerDied","Data":"efff3d94cefbac178494d9bf47ee2c971add39c24063563bef1afd91b46e805e"} Oct 30 22:04:34.498522 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.498482 4526 scope.go:122] "RemoveContainer" containerID="a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651" Oct 30 22:04:34.498829 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:34.498811 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651\": container with ID starting with a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651 not found: ID does not exist" containerID="a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651" Oct 30 22:04:34.498890 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.498835 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651"} err="failed to get container status \"a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651\": rpc error: code = NotFound desc = could not find container \"a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651\": container with ID starting with a7db8d1aa5bd946f299bbdfefa81dd533ce4e14c07cc5a9a8343481b21976651 not found: ID does not exist" Oct 30 22:04:34.498890 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.498851 4526 scope.go:122] "RemoveContainer" containerID="4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8" Oct 30 22:04:34.507537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.507516 4526 scope.go:122] "RemoveContainer" containerID="4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8" Oct 30 22:04:34.507837 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:34.507818 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8\": container with ID starting with 4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8 not found: ID does not exist" containerID="4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8" Oct 30 22:04:34.507895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.507847 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8"} err="failed to get container status \"4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8\": rpc error: code = NotFound desc = could not find container \"4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8\": container with ID starting with 4e082bcfa94742ed7a0319a27d22818099492b3e61881e2564b83a31ab2172a8 not found: ID does not exist" Oct 30 22:04:34.514173 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514149 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-home\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-home\") pod \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " Oct 30 22:04:34.514253 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514210 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-model-cache\") pod \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " Oct 30 22:04:34.514253 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514246 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-dshm\") pod \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " Oct 30 22:04:34.514341 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514270 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32765e94-ffd5-4160-b1bd-0636b0f5ac86-kube-api-access-6ddn5\" (UniqueName: \"kubernetes.io/projected/32765e94-ffd5-4160-b1bd-0636b0f5ac86-kube-api-access-6ddn5\") pod \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " Oct 30 22:04:34.514341 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514328 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tls-certs\" (UniqueName: \"kubernetes.io/secret/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tls-certs\") pod \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " Oct 30 22:04:34.514427 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514397 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tmp-dir\") pod \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\" (UID: \"32765e94-ffd5-4160-b1bd-0636b0f5ac86\") " Oct 30 22:04:34.514478 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514420 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-home" pod "32765e94-ffd5-4160-b1bd-0636b0f5ac86" (UID: "32765e94-ffd5-4160-b1bd-0636b0f5ac86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.514549 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514531 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-model-cache" pod "32765e94-ffd5-4160-b1bd-0636b0f5ac86" (UID: "32765e94-ffd5-4160-b1bd-0636b0f5ac86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.514686 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514662 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tmp-dir" pod "32765e94-ffd5-4160-b1bd-0636b0f5ac86" (UID: "32765e94-ffd5-4160-b1bd-0636b0f5ac86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.514822 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514801 4526 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tmp-dir\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.514822 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514825 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr9k6\" (UniqueName: \"kubernetes.io/projected/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-kube-api-access-pr9k6\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.515179 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514835 4526 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-home\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.515179 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514844 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.515179 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514852 4526 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-model-cache\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.515179 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.514862 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.516600 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.516582 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-dshm" pod "32765e94-ffd5-4160-b1bd-0636b0f5ac86" (UID: "32765e94-ffd5-4160-b1bd-0636b0f5ac86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.516683 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.516653 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32765e94-ffd5-4160-b1bd-0636b0f5ac86-kube-api-access-6ddn5" pod "32765e94-ffd5-4160-b1bd-0636b0f5ac86" (UID: "32765e94-ffd5-4160-b1bd-0636b0f5ac86"). InnerVolumeSpecName "kube-api-access-6ddn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:34.516683 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.516655 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tls-certs" pod "32765e94-ffd5-4160-b1bd-0636b0f5ac86" (UID: "32765e94-ffd5-4160-b1bd-0636b0f5ac86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:34.616117 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.616091 4526 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32765e94-ffd5-4160-b1bd-0636b0f5ac86-dshm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.616233 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.616122 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ddn5\" (UniqueName: \"kubernetes.io/projected/32765e94-ffd5-4160-b1bd-0636b0f5ac86-kube-api-access-6ddn5\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.616233 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.616136 4526 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32765e94-ffd5-4160-b1bd-0636b0f5ac86-tls-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.876190 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.876101 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m"] Oct 30 22:04:34.878148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.878126 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m"] Oct 30 22:04:34.985428 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.985390 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj"] Oct 30 22:04:34.991078 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:34.991048 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/simulator-w2-446ca2-kserve-7467554bb9-h4rfj"] Oct 30 22:04:35.324186 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:35.324133 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/distinct-2-w2-446ca2-kserve-7499f65988-s954m" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 22:04:35.984371 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:35.984337 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" path="/var/lib/kubelet/pods/32765e94-ffd5-4160-b1bd-0636b0f5ac86/volumes" Oct 30 22:04:35.984790 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:35.984775 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" path="/var/lib/kubelet/pods/cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0/volumes" Oct 30 22:04:39.025760 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:39.025721 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" podUID="6b9e0bc4-bf67-413f-a580-974404f77720" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:39.026155 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:39.025726 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" podUID="79acc862-069d-4c9e-9759-535067b6e745" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:42.218181 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:42.218150 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg"] Oct 30 22:04:42.267251 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:42.267216 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-446ca2-8ljrg"] Oct 30 22:04:42.477678 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:42.477588 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:04:43.985001 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:43.984962 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1c6200-5dd2-437c-b3b3-18b919ef7496" path="/var/lib/kubelet/pods/0b1c6200-5dd2-437c-b3b3-18b919ef7496/volumes" Oct 30 22:04:44.073249 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:44.073212 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" podUID="06dfcdda-bbd5-4ebb-9829-f6c23e06982f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:04:47.182343 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.181708 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 22:04:47.244603 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.244574 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79acc862-069d-4c9e-9759-535067b6e745-kube-api-access-8wv5w\" (UniqueName: \"kubernetes.io/projected/79acc862-069d-4c9e-9759-535067b6e745-kube-api-access-8wv5w\") pod \"79acc862-069d-4c9e-9759-535067b6e745\" (UID: \"79acc862-069d-4c9e-9759-535067b6e745\") " Oct 30 22:04:47.244819 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.244735 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79acc862-069d-4c9e-9759-535067b6e745-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acc862-069d-4c9e-9759-535067b6e745-plugins-config-volume\") pod \"79acc862-069d-4c9e-9759-535067b6e745\" (UID: \"79acc862-069d-4c9e-9759-535067b6e745\") " Oct 30 22:04:47.245102 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.245074 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79acc862-069d-4c9e-9759-535067b6e745-plugins-config-volume" pod "79acc862-069d-4c9e-9759-535067b6e745" (UID: "79acc862-069d-4c9e-9759-535067b6e745"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:47.247909 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.247884 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79acc862-069d-4c9e-9759-535067b6e745-kube-api-access-8wv5w" pod "79acc862-069d-4c9e-9759-535067b6e745" (UID: "79acc862-069d-4c9e-9759-535067b6e745"). InnerVolumeSpecName "kube-api-access-8wv5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:47.278008 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.277982 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 22:04:47.345374 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.345339 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9e0bc4-bf67-413f-a580-974404f77720-kube-api-access-7dvpd\" (UniqueName: \"kubernetes.io/projected/6b9e0bc4-bf67-413f-a580-974404f77720-kube-api-access-7dvpd\") pod \"6b9e0bc4-bf67-413f-a580-974404f77720\" (UID: \"6b9e0bc4-bf67-413f-a580-974404f77720\") " Oct 30 22:04:47.345606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.345394 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9e0bc4-bf67-413f-a580-974404f77720-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e0bc4-bf67-413f-a580-974404f77720-plugins-config-volume\") pod \"6b9e0bc4-bf67-413f-a580-974404f77720\" (UID: \"6b9e0bc4-bf67-413f-a580-974404f77720\") " Oct 30 22:04:47.345688 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.345604 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79acc862-069d-4c9e-9759-535067b6e745-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:47.345688 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.345618 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wv5w\" (UniqueName: \"kubernetes.io/projected/79acc862-069d-4c9e-9759-535067b6e745-kube-api-access-8wv5w\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:47.345820 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.345797 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9e0bc4-bf67-413f-a580-974404f77720-plugins-config-volume" pod "6b9e0bc4-bf67-413f-a580-974404f77720" (UID: "6b9e0bc4-bf67-413f-a580-974404f77720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:47.347760 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.347738 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9e0bc4-bf67-413f-a580-974404f77720-kube-api-access-7dvpd" pod "6b9e0bc4-bf67-413f-a580-974404f77720" (UID: "6b9e0bc4-bf67-413f-a580-974404f77720"). InnerVolumeSpecName "kube-api-access-7dvpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:47.446258 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.446219 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9e0bc4-bf67-413f-a580-974404f77720-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:47.446258 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.446247 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvpd\" (UniqueName: \"kubernetes.io/projected/6b9e0bc4-bf67-413f-a580-974404f77720-kube-api-access-7dvpd\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:47.562930 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.562899 4526 generic.go:356] "Generic (PLEG): container finished" podID="79acc862-069d-4c9e-9759-535067b6e745" containerID="398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6" exitCode=0 Oct 30 22:04:47.563121 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.562956 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" Oct 30 22:04:47.563121 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.562985 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" event={"ID":"79acc862-069d-4c9e-9759-535067b6e745","Type":"ContainerDied","Data":"398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6"} Oct 30 22:04:47.563121 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.563045 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9" event={"ID":"79acc862-069d-4c9e-9759-535067b6e745","Type":"ContainerDied","Data":"fc7db092ee0ff66c8efc46463bb76e6d0d507d9ce9121ef492222a3e45183f1d"} Oct 30 22:04:47.563121 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.563069 4526 scope.go:122] "RemoveContainer" containerID="398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6" Oct 30 22:04:47.565096 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.564681 4526 generic.go:356] "Generic (PLEG): container finished" podID="6b9e0bc4-bf67-413f-a580-974404f77720" containerID="6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e" exitCode=0 Oct 30 22:04:47.565096 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.564711 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" event={"ID":"6b9e0bc4-bf67-413f-a580-974404f77720","Type":"ContainerDied","Data":"6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e"} Oct 30 22:04:47.565096 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.564744 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" Oct 30 22:04:47.565096 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.564920 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984" event={"ID":"6b9e0bc4-bf67-413f-a580-974404f77720","Type":"ContainerDied","Data":"33ba5438cf2795649d90b47c0c30ac96dd702bbc40adab000828c6c3ecc157ff"} Oct 30 22:04:47.573114 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.572948 4526 scope.go:122] "RemoveContainer" containerID="398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6" Oct 30 22:04:47.573243 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:47.573221 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6\": container with ID starting with 398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6 not found: ID does not exist" containerID="398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6" Oct 30 22:04:47.573314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.573245 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6"} err="failed to get container status \"398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6\": rpc error: code = NotFound desc = could not find container \"398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6\": container with ID starting with 398aae6a5597ec9f0306c5af30636b1ea59dbc1426fd7ae62669a373242d6db6 not found: ID does not exist" Oct 30 22:04:47.573314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.573261 4526 scope.go:122] "RemoveContainer" containerID="6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e" Oct 30 22:04:47.581917 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.581899 4526 scope.go:122] "RemoveContainer" containerID="6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e" Oct 30 22:04:47.582222 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:47.582205 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e\": container with ID starting with 6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e not found: ID does not exist" containerID="6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e" Oct 30 22:04:47.582280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.582230 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e"} err="failed to get container status \"6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e\": rpc error: code = NotFound desc = could not find container \"6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e\": container with ID starting with 6994c1351ed9b7fbd25f6e3b5c975b6c1671b6eccf78506af71cd196b04ffa5e not found: ID does not exist" Oct 30 22:04:47.725782 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.725738 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9"] Oct 30 22:04:47.731741 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.731698 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-07fa8e-c95f9979c-h5hf9"] Oct 30 22:04:47.841204 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.841109 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984"] Oct 30 22:04:47.844327 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.844293 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-07fa8e-769597d555-kl984"] Oct 30 22:04:47.983052 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.983020 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9e0bc4-bf67-413f-a580-974404f77720" path="/var/lib/kubelet/pods/6b9e0bc4-bf67-413f-a580-974404f77720/volumes" Oct 30 22:04:47.983399 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:47.983386 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79acc862-069d-4c9e-9759-535067b6e745" path="/var/lib/kubelet/pods/79acc862-069d-4c9e-9759-535067b6e745/volumes" Oct 30 22:04:51.339121 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.339081 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv"] Oct 30 22:04:51.388812 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.388773 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-07fa8e-grwxv"] Oct 30 22:04:51.493956 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.493933 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 22:04:51.585589 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.585551 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-plugins-config-volume\") pod \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\" (UID: \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\") " Oct 30 22:04:51.585589 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.585595 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-kube-api-access-4cfpw\" (UniqueName: \"kubernetes.io/projected/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-kube-api-access-4cfpw\") pod \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\" (UID: \"06dfcdda-bbd5-4ebb-9829-f6c23e06982f\") " Oct 30 22:04:51.586018 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.585980 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-plugins-config-volume" pod "06dfcdda-bbd5-4ebb-9829-f6c23e06982f" (UID: "06dfcdda-bbd5-4ebb-9829-f6c23e06982f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:51.587999 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.587976 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-kube-api-access-4cfpw" pod "06dfcdda-bbd5-4ebb-9829-f6c23e06982f" (UID: "06dfcdda-bbd5-4ebb-9829-f6c23e06982f"). InnerVolumeSpecName "kube-api-access-4cfpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:51.592632 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.592608 4526 generic.go:356] "Generic (PLEG): container finished" podID="06dfcdda-bbd5-4ebb-9829-f6c23e06982f" containerID="59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0" exitCode=0 Oct 30 22:04:51.592757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.592665 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" Oct 30 22:04:51.592757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.592685 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" event={"ID":"06dfcdda-bbd5-4ebb-9829-f6c23e06982f","Type":"ContainerDied","Data":"59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0"} Oct 30 22:04:51.592868 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.592768 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq" event={"ID":"06dfcdda-bbd5-4ebb-9829-f6c23e06982f","Type":"ContainerDied","Data":"3ea485008fc99e34ff5824f205b7322eeddb8965ecd48a150684a8e5916a81e5"} Oct 30 22:04:51.592868 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.592791 4526 scope.go:122] "RemoveContainer" containerID="59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0" Oct 30 22:04:51.601996 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.601975 4526 scope.go:122] "RemoveContainer" containerID="59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0" Oct 30 22:04:51.602964 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:04:51.602943 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0\": container with ID starting with 59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0 not found: ID does not exist" containerID="59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0" Oct 30 22:04:51.603055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.602976 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0"} err="failed to get container status \"59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0\": rpc error: code = NotFound desc = could not find container \"59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0\": container with ID starting with 59b5f50c3a036db3c4ecb253d93f6bdaf62105899128848b33b6cea3263511b0 not found: ID does not exist" Oct 30 22:04:51.687249 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.687212 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:51.687249 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.687244 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cfpw\" (UniqueName: \"kubernetes.io/projected/06dfcdda-bbd5-4ebb-9829-f6c23e06982f-kube-api-access-4cfpw\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:51.720968 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.720935 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq"] Oct 30 22:04:51.725321 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.725288 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-446ca2-5f4d7f56f5-znhbq"] Oct 30 22:04:51.984259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.984172 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06dfcdda-bbd5-4ebb-9829-f6c23e06982f" path="/var/lib/kubelet/pods/06dfcdda-bbd5-4ebb-9829-f6c23e06982f/volumes" Oct 30 22:04:51.984742 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:51.984721 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d94dc2f-5efb-4b3b-a869-8e415fdcac49" path="/var/lib/kubelet/pods/7d94dc2f-5efb-4b3b-a869-8e415fdcac49/volumes" Oct 30 22:04:52.690380 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:52.690350 4526 scope.go:122] "RemoveContainer" containerID="6fceeeeaa3e147a26630a2f40657c67204dfce1b8d11aea38c343f70fedfc302" Oct 30 22:04:57.883548 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:57.883521 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc"] Oct 30 22:04:57.883954 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:04:57.883775 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" containerID="cri-o://eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66" gracePeriod=30 Oct 30 22:05:03.132895 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.132869 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 22:05:03.300054 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300022 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-credential-socket\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300244 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300063 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-kube-api-access-w75g2\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-kube-api-access-w75g2\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300244 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300096 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-certs\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300244 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300122 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-data\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300244 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300157 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda6d95a-c501-4387-b6c8-694af23150e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda6d95a-c501-4387-b6c8-694af23150e6-istiod-ca-cert\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300244 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300212 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cda6d95a-c501-4387-b6c8-694af23150e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda6d95a-c501-4387-b6c8-694af23150e6-istio-podinfo\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300478 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300255 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-istio-token\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-istio-token\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300478 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300360 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-socket\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300478 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300394 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-envoy\") pod \"cda6d95a-c501-4387-b6c8-694af23150e6\" (UID: \"cda6d95a-c501-4387-b6c8-694af23150e6\") " Oct 30 22:05:03.300478 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300422 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-credential-socket" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.300808 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300773 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.300808 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.300780 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda6d95a-c501-4387-b6c8-694af23150e6-istiod-ca-cert" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:03.301182 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.301157 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-socket" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.301288 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.301176 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-certs" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.301618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.301405 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-data" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.303608 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.303354 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cda6d95a-c501-4387-b6c8-694af23150e6-istio-podinfo" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:05:03.303608 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.303525 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-kube-api-access-w75g2" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "kube-api-access-w75g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:03.304008 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.303947 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-envoy" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.304008 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.303990 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-istio-token" pod "cda6d95a-c501-4387-b6c8-694af23150e6" (UID: "cda6d95a-c501-4387-b6c8-694af23150e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:03.402009 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.401971 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.402009 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.402001 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.402009 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.402011 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w75g2\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-kube-api-access-w75g2\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.402009 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.402020 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.402299 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.402028 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda6d95a-c501-4387-b6c8-694af23150e6-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.402299 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.402038 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda6d95a-c501-4387-b6c8-694af23150e6-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.402299 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.402046 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda6d95a-c501-4387-b6c8-694af23150e6-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.402299 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.402054 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda6d95a-c501-4387-b6c8-694af23150e6-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.657318 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.657221 4526 generic.go:356] "Generic (PLEG): container finished" podID="cda6d95a-c501-4387-b6c8-694af23150e6" containerID="eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66" exitCode=0 Oct 30 22:05:03.657318 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.657283 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" Oct 30 22:05:03.657597 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.657316 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" event={"ID":"cda6d95a-c501-4387-b6c8-694af23150e6","Type":"ContainerDied","Data":"eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66"} Oct 30 22:05:03.657597 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.657376 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc" event={"ID":"cda6d95a-c501-4387-b6c8-694af23150e6","Type":"ContainerDied","Data":"db17289e0c4df6ff67f78cf2b1fb9e884b3ff68326941275e74b6428e13bceb7"} Oct 30 22:05:03.657597 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.657402 4526 scope.go:122] "RemoveContainer" containerID="eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66" Oct 30 22:05:03.666846 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.666826 4526 scope.go:122] "RemoveContainer" containerID="7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c" Oct 30 22:05:03.675582 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.675561 4526 scope.go:122] "RemoveContainer" containerID="eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66" Oct 30 22:05:03.675933 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:05:03.675913 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66\": container with ID starting with eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66 not found: ID does not exist" containerID="eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66" Oct 30 22:05:03.676012 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.675946 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66"} err="failed to get container status \"eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66\": rpc error: code = NotFound desc = could not find container \"eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66\": container with ID starting with eb5e0366c94ac2c47fd147fa7314a9b77b5e04f60538e1b188bacf37e84dba66 not found: ID does not exist" Oct 30 22:05:03.676012 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.675975 4526 scope.go:122] "RemoveContainer" containerID="7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c" Oct 30 22:05:03.676258 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:05:03.676240 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c\": container with ID starting with 7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c not found: ID does not exist" containerID="7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c" Oct 30 22:05:03.676315 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.676264 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c"} err="failed to get container status \"7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c\": rpc error: code = NotFound desc = could not find container \"7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c\": container with ID starting with 7f29f64c215bccb521cb72df420af79781f4cdfeb9ba16091987b68ff526641c not found: ID does not exist" Oct 30 22:05:03.820631 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.820588 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc"] Oct 30 22:05:03.867241 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.867195 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-d5prc"] Oct 30 22:05:03.985053 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:03.984968 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" path="/var/lib/kubelet/pods/cda6d95a-c501-4387-b6c8-694af23150e6/volumes" Oct 30 22:05:12.478104 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:05:12.478066 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:05:38.217804 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.217719 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf"] Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218092 4526 cpu_manager.go:414] "containerMap: removing container" podUID="06dfcdda-bbd5-4ebb-9829-f6c23e06982f" containerName="payload-pre-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218103 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dfcdda-bbd5-4ebb-9829-f6c23e06982f" containerName="payload-pre-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218114 4526 cpu_manager.go:414] "containerMap: removing container" podUID="79acc862-069d-4c9e-9759-535067b6e745" containerName="payload-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218120 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79acc862-069d-4c9e-9759-535067b6e745" containerName="payload-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218138 4526 cpu_manager.go:414] "containerMap: removing container" podUID="57944f20-befb-4a33-8288-1aa2b87024de" containerName="revoke-keys" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218143 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57944f20-befb-4a33-8288-1aa2b87024de" containerName="revoke-keys" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218152 4526 cpu_manager.go:414] "containerMap: removing container" podUID="af111955-96d6-46e2-8103-2e4f51b8a622" containerName="payload-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218157 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af111955-96d6-46e2-8103-2e4f51b8a622" containerName="payload-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218164 4526 cpu_manager.go:414] "containerMap: removing container" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218168 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218180 4526 cpu_manager.go:414] "containerMap: removing container" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218184 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218190 4526 cpu_manager.go:414] "containerMap: removing container" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218196 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218209 4526 cpu_manager.go:414] "containerMap: removing container" podUID="6b9e0bc4-bf67-413f-a580-974404f77720" containerName="payload-pre-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218214 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9e0bc4-bf67-413f-a580-974404f77720" containerName="payload-pre-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218225 4526 cpu_manager.go:414] "containerMap: removing container" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218229 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218240 4526 cpu_manager.go:414] "containerMap: removing container" podUID="b1df5bca-fc84-4bec-bd46-cd7b804234de" containerName="revoke-keys" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218245 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1df5bca-fc84-4bec-bd46-cd7b804234de" containerName="revoke-keys" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218307 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="32765e94-ffd5-4160-b1bd-0636b0f5ac86" containerName="main" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218315 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218321 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9e0bc4-bf67-413f-a580-974404f77720" containerName="payload-pre-processing" Oct 30 22:05:38.218314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218327 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd88f814-b3c3-48f9-a4ee-e9c7bafb92b0" containerName="main" Oct 30 22:05:38.219148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218336 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1df5bca-fc84-4bec-bd46-cd7b804234de" containerName="revoke-keys" Oct 30 22:05:38.219148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218343 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dfcdda-bbd5-4ebb-9829-f6c23e06982f" containerName="payload-pre-processing" Oct 30 22:05:38.219148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218348 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="79acc862-069d-4c9e-9759-535067b6e745" containerName="payload-processing" Oct 30 22:05:38.219148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218357 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="57944f20-befb-4a33-8288-1aa2b87024de" containerName="revoke-keys" Oct 30 22:05:38.219148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218365 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="af111955-96d6-46e2-8103-2e4f51b8a622" containerName="payload-processing" Oct 30 22:05:38.219148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.218371 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda6d95a-c501-4387-b6c8-694af23150e6" containerName="istio-proxy" Oct 30 22:05:38.221503 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.221471 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.276163 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.276130 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:38.276163 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.276151 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:38.279037 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.279009 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf"] Oct 30 22:05:38.325479 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.325447 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:38.325479 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.325469 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:05:38.336167 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.336137 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxrsm\" (UniqueName: \"kubernetes.io/projected/59fd3368-cd61-40ed-bbf6-d78956bf8448-kube-api-access-qxrsm\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.336413 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.336383 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.336544 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.336445 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.437921 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.437876 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.438107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.437930 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.438107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.438005 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxrsm\" (UniqueName: \"kubernetes.io/projected/59fd3368-cd61-40ed-bbf6-d78956bf8448-kube-api-access-qxrsm\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.438588 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.438558 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.440527 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.440477 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.494824 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.494751 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxrsm\" (UniqueName: \"kubernetes.io/projected/59fd3368-cd61-40ed-bbf6-d78956bf8448-kube-api-access-qxrsm\") pod \"maas-api-revoke-sub-85237e45-9g6hf\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.533926 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.533894 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:38.904004 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:38.903977 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf"] Oct 30 22:05:38.906793 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:05:38.906761 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fd3368_cd61_40ed_bbf6_d78956bf8448.slice/crio-8bdb36d52b745fa977bee857cfc3f04c11ec1fff34ce3802e614d35dae442e77 WatchSource:0}: Error finding container 8bdb36d52b745fa977bee857cfc3f04c11ec1fff34ce3802e614d35dae442e77: Status 404 returned error can't find the container with id 8bdb36d52b745fa977bee857cfc3f04c11ec1fff34ce3802e614d35dae442e77 Oct 30 22:05:39.845962 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:39.845930 4526 generic.go:356] "Generic (PLEG): container finished" podID="59fd3368-cd61-40ed-bbf6-d78956bf8448" containerID="fb168ba94119cd3052e9400a92f645e4df864b7f753c5a8145fffb917a5561b8" exitCode=0 Oct 30 22:05:39.846396 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:39.846020 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" event={"ID":"59fd3368-cd61-40ed-bbf6-d78956bf8448","Type":"ContainerDied","Data":"fb168ba94119cd3052e9400a92f645e4df864b7f753c5a8145fffb917a5561b8"} Oct 30 22:05:39.846396 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:39.846086 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" event={"ID":"59fd3368-cd61-40ed-bbf6-d78956bf8448","Type":"ContainerStarted","Data":"8bdb36d52b745fa977bee857cfc3f04c11ec1fff34ce3802e614d35dae442e77"} Oct 30 22:05:40.985078 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:40.985054 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:41.161243 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.161156 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-xks-ca\") pod \"59fd3368-cd61-40ed-bbf6-d78956bf8448\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " Oct 30 22:05:41.161426 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.161278 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-openshift-ca\") pod \"59fd3368-cd61-40ed-bbf6-d78956bf8448\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " Oct 30 22:05:41.161426 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.161336 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fd3368-cd61-40ed-bbf6-d78956bf8448-kube-api-access-qxrsm\" (UniqueName: \"kubernetes.io/projected/59fd3368-cd61-40ed-bbf6-d78956bf8448-kube-api-access-qxrsm\") pod \"59fd3368-cd61-40ed-bbf6-d78956bf8448\" (UID: \"59fd3368-cd61-40ed-bbf6-d78956bf8448\") " Oct 30 22:05:41.161686 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.161659 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-openshift-ca" pod "59fd3368-cd61-40ed-bbf6-d78956bf8448" (UID: "59fd3368-cd61-40ed-bbf6-d78956bf8448"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:41.163537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.163507 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-xks-ca" pod "59fd3368-cd61-40ed-bbf6-d78956bf8448" (UID: "59fd3368-cd61-40ed-bbf6-d78956bf8448"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:41.163634 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.163561 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fd3368-cd61-40ed-bbf6-d78956bf8448-kube-api-access-qxrsm" pod "59fd3368-cd61-40ed-bbf6-d78956bf8448" (UID: "59fd3368-cd61-40ed-bbf6-d78956bf8448"). InnerVolumeSpecName "kube-api-access-qxrsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:41.262469 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.262434 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:41.262469 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.262461 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59fd3368-cd61-40ed-bbf6-d78956bf8448-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:41.262469 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.262471 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxrsm\" (UniqueName: \"kubernetes.io/projected/59fd3368-cd61-40ed-bbf6-d78956bf8448-kube-api-access-qxrsm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:41.857588 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.857557 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" Oct 30 22:05:41.857588 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.857579 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf" event={"ID":"59fd3368-cd61-40ed-bbf6-d78956bf8448","Type":"ContainerDied","Data":"8bdb36d52b745fa977bee857cfc3f04c11ec1fff34ce3802e614d35dae442e77"} Oct 30 22:05:41.857810 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:41.857626 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bdb36d52b745fa977bee857cfc3f04c11ec1fff34ce3802e614d35dae442e77" Oct 30 22:05:42.479008 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:05:42.478970 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:05:52.762127 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:52.762096 4526 scope.go:122] "RemoveContainer" containerID="695fc4c237e0449d3ce26e9644d3652d4a7c3bff5853b82e04a5a42f370b47d8" Oct 30 22:05:52.770978 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:52.770960 4526 scope.go:122] "RemoveContainer" containerID="cd67142fc7924a63958c4e59fdfe5f69b2ac5f799e2014b59b2a75cc44d5dec2" Oct 30 22:05:52.778867 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:52.778847 4526 scope.go:122] "RemoveContainer" containerID="464cafa32093cddea1b0fa3c8776c436bc40aee95019587cb11245cb5bbccc64" Oct 30 22:05:52.787592 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:05:52.787561 4526 scope.go:122] "RemoveContainer" containerID="1d5d50697dd4a3ace694573a6688246f81ec0d763f511c83439b5343157debc0" Oct 30 22:06:12.478306 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:06:12.478264 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:06:42.477201 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:06:42.477155 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:06:53.044811 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:06:53.044774 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp"] Oct 30 22:06:53.094134 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:06:53.094099 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-369374b7-tdkgp"] Oct 30 22:06:53.983656 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:06:53.983627 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ca2f99-7eb7-4bba-998d-1907101ea3b4" path="/var/lib/kubelet/pods/73ca2f99-7eb7-4bba-998d-1907101ea3b4/volumes" Oct 30 22:06:56.189702 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:06:56.189653 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:06:56.190133 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:06:56.189735 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:07:02.672848 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.672764 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs"] Oct 30 22:07:02.673239 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.673145 4526 cpu_manager.go:414] "containerMap: removing container" podUID="59fd3368-cd61-40ed-bbf6-d78956bf8448" containerName="revoke-keys" Oct 30 22:07:02.673239 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.673156 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fd3368-cd61-40ed-bbf6-d78956bf8448" containerName="revoke-keys" Oct 30 22:07:02.673239 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.673218 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fd3368-cd61-40ed-bbf6-d78956bf8448" containerName="revoke-keys" Oct 30 22:07:02.676347 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.676330 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.733528 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.733467 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs"] Oct 30 22:07:02.777655 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.777628 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:02.777655 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.777653 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:02.781346 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.781297 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:02.781475 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.781370 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:07:02.799132 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.799104 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnqd\" (UniqueName: \"kubernetes.io/projected/dd7d9181-6a3e-414a-b580-e04f2d713fe6-kube-api-access-sbnqd\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.799321 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.799151 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.799321 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.799244 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.900653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.900612 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnqd\" (UniqueName: \"kubernetes.io/projected/dd7d9181-6a3e-414a-b580-e04f2d713fe6-kube-api-access-sbnqd\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.900864 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.900678 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.900864 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.900716 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.901455 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.901435 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.903184 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.903161 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.957250 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.957171 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnqd\" (UniqueName: \"kubernetes.io/projected/dd7d9181-6a3e-414a-b580-e04f2d713fe6-kube-api-access-sbnqd\") pod \"maas-api-revoke-sub-c1e271a9-dtpgs\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:02.988588 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:02.988549 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:03.361383 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:03.361349 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs"] Oct 30 22:07:03.364717 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:07:03.364687 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7d9181_6a3e_414a_b580_e04f2d713fe6.slice/crio-e5d23efbec954e946238feb6975d63c353562dc235ea077093793a91196c4582 WatchSource:0}: Error finding container e5d23efbec954e946238feb6975d63c353562dc235ea077093793a91196c4582: Status 404 returned error can't find the container with id e5d23efbec954e946238feb6975d63c353562dc235ea077093793a91196c4582 Oct 30 22:07:04.257709 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:04.257667 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" event={"ID":"dd7d9181-6a3e-414a-b580-e04f2d713fe6","Type":"ContainerStarted","Data":"2eeedbf05ca9fa07ad04f96e91b061191b32f9780a1ceaafeb9bec1a3162936a"} Oct 30 22:07:04.258111 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:04.257740 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" event={"ID":"dd7d9181-6a3e-414a-b580-e04f2d713fe6","Type":"ContainerStarted","Data":"e5d23efbec954e946238feb6975d63c353562dc235ea077093793a91196c4582"} Oct 30 22:07:05.264864 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:05.264832 4526 generic.go:356] "Generic (PLEG): container finished" podID="dd7d9181-6a3e-414a-b580-e04f2d713fe6" containerID="2eeedbf05ca9fa07ad04f96e91b061191b32f9780a1ceaafeb9bec1a3162936a" exitCode=0 Oct 30 22:07:05.265249 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:05.264921 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" event={"ID":"dd7d9181-6a3e-414a-b580-e04f2d713fe6","Type":"ContainerDied","Data":"2eeedbf05ca9fa07ad04f96e91b061191b32f9780a1ceaafeb9bec1a3162936a"} Oct 30 22:07:06.406125 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.406097 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:06.534784 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.534749 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-xks-ca\") pod \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " Oct 30 22:07:06.535003 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.534837 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7d9181-6a3e-414a-b580-e04f2d713fe6-kube-api-access-sbnqd\" (UniqueName: \"kubernetes.io/projected/dd7d9181-6a3e-414a-b580-e04f2d713fe6-kube-api-access-sbnqd\") pod \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " Oct 30 22:07:06.535003 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.534867 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-openshift-ca\") pod \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\" (UID: \"dd7d9181-6a3e-414a-b580-e04f2d713fe6\") " Oct 30 22:07:06.535317 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.535272 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-openshift-ca" pod "dd7d9181-6a3e-414a-b580-e04f2d713fe6" (UID: "dd7d9181-6a3e-414a-b580-e04f2d713fe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:06.536961 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.536934 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-xks-ca" pod "dd7d9181-6a3e-414a-b580-e04f2d713fe6" (UID: "dd7d9181-6a3e-414a-b580-e04f2d713fe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:06.537204 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.537189 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7d9181-6a3e-414a-b580-e04f2d713fe6-kube-api-access-sbnqd" pod "dd7d9181-6a3e-414a-b580-e04f2d713fe6" (UID: "dd7d9181-6a3e-414a-b580-e04f2d713fe6"). InnerVolumeSpecName "kube-api-access-sbnqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:06.636368 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.636331 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:06.636368 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.636363 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnqd\" (UniqueName: \"kubernetes.io/projected/dd7d9181-6a3e-414a-b580-e04f2d713fe6-kube-api-access-sbnqd\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:06.636368 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:06.636374 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7d9181-6a3e-414a-b580-e04f2d713fe6-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:07.275881 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:07.275849 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" Oct 30 22:07:07.276088 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:07.275890 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs" event={"ID":"dd7d9181-6a3e-414a-b580-e04f2d713fe6","Type":"ContainerDied","Data":"e5d23efbec954e946238feb6975d63c353562dc235ea077093793a91196c4582"} Oct 30 22:07:07.276088 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:07.275944 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5d23efbec954e946238feb6975d63c353562dc235ea077093793a91196c4582" Oct 30 22:07:08.047810 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:08.047777 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89"] Oct 30 22:07:08.097903 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:08.097866 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960b5194-7gs89"] Oct 30 22:07:09.983900 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:09.983872 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df39b071-bb0c-4c45-9dd7-4d86a2501817" path="/var/lib/kubelet/pods/df39b071-bb0c-4c45-9dd7-4d86a2501817/volumes" Oct 30 22:07:12.477509 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:07:12.477457 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:07:18.368537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.368498 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq"] Oct 30 22:07:18.369081 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.369062 4526 cpu_manager.go:414] "containerMap: removing container" podUID="dd7d9181-6a3e-414a-b580-e04f2d713fe6" containerName="revoke-keys" Oct 30 22:07:18.369160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.369083 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7d9181-6a3e-414a-b580-e04f2d713fe6" containerName="revoke-keys" Oct 30 22:07:18.369244 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.369228 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7d9181-6a3e-414a-b580-e04f2d713fe6" containerName="revoke-keys" Oct 30 22:07:18.372762 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.372741 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.418026 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.417999 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4"] Oct 30 22:07:18.421847 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.421827 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.478419 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.478382 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq"] Oct 30 22:07:18.528576 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.528544 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4"] Oct 30 22:07:18.544288 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544252 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.544288 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544289 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.544517 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544319 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4fc\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-kube-api-access-vf4fc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.544517 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544388 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.544603 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544512 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fn9k\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-kube-api-access-6fn9k\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.544603 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544568 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.544603 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544597 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.544711 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544672 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.544711 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544692 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9630be5-74c9-4c96-aee6-459cc0c2943b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.544781 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544765 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.544827 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544810 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.544864 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544844 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9551403-60a7-4a70-a8e5-aa86043b3185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.544899 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544885 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.544957 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544937 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.544994 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.544972 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.545060 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.545048 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.545096 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.545088 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.545132 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.545116 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.645922 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.645840 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.645922 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.645879 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.645922 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.645896 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9551403-60a7-4a70-a8e5-aa86043b3185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.645922 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.645914 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.645930 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.645959 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.645994 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646048 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646075 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646101 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646127 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646165 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4fc\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-kube-api-access-vf4fc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646192 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646236 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fn9k\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-kube-api-access-6fn9k\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646292 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646316 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646371 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646375 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646388 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646396 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9630be5-74c9-4c96-aee6-459cc0c2943b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646731 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9551403-60a7-4a70-a8e5-aa86043b3185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.646794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646732 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.647094 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646822 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.647094 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.646919 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.647094 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.647069 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.647263 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.647179 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.647540 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.647428 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.647666 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.647648 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9630be5-74c9-4c96-aee6-459cc0c2943b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.648650 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.648629 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.648927 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.648911 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.649306 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.649286 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.649651 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.649635 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.703970 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.703937 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.704131 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.704067 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fn9k\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-kube-api-access-6fn9k\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.704293 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.704268 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4fc\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-kube-api-access-vf4fc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qxjl4\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.704293 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.704287 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-f7ssq\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:18.734445 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.734421 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:18.986098 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:18.986011 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:19.160273 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.160248 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4"] Oct 30 22:07:19.162474 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:07:19.162442 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9630be5_74c9_4c96_aee6_459cc0c2943b.slice/crio-c25819e6cbfd06a23a5c6c13e8dee09c24e3371301ed7ee3e60fcfc769930a6f WatchSource:0}: Error finding container c25819e6cbfd06a23a5c6c13e8dee09c24e3371301ed7ee3e60fcfc769930a6f: Status 404 returned error can't find the container with id c25819e6cbfd06a23a5c6c13e8dee09c24e3371301ed7ee3e60fcfc769930a6f Oct 30 22:07:19.164718 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.164687 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.164831 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.164762 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.164831 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.164812 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.335871 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.335833 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" event={"ID":"e9630be5-74c9-4c96-aee6-459cc0c2943b","Type":"ContainerStarted","Data":"8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968"} Oct 30 22:07:19.336053 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.335893 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" event={"ID":"e9630be5-74c9-4c96-aee6-459cc0c2943b","Type":"ContainerStarted","Data":"c25819e6cbfd06a23a5c6c13e8dee09c24e3371301ed7ee3e60fcfc769930a6f"} Oct 30 22:07:19.363606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.363578 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq"] Oct 30 22:07:19.366283 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:07:19.366254 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9551403_60a7_4a70_a8e5_aa86043b3185.slice/crio-2aa7a709f6e3debddbaa017c2d73f1aa501556704bfffa70480648bef08ad796 WatchSource:0}: Error finding container 2aa7a709f6e3debddbaa017c2d73f1aa501556704bfffa70480648bef08ad796: Status 404 returned error can't find the container with id 2aa7a709f6e3debddbaa017c2d73f1aa501556704bfffa70480648bef08ad796 Oct 30 22:07:19.369026 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.368989 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.369345 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.369070 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.369345 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.369107 4526 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.454304 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.454242 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" podStartSLOduration=1.454223266 podStartE2EDuration="1.454223266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:19.452311629 +0000 UTC m=+3029.585684828" watchObservedRunningTime="2026-10-30 22:07:19.454223266 +0000 UTC m=+3029.587596465" Oct 30 22:07:19.735587 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:19.735465 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:20.350325 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:20.350280 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" event={"ID":"f9551403-60a7-4a70-a8e5-aa86043b3185","Type":"ContainerStarted","Data":"8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3"} Oct 30 22:07:20.350575 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:20.350335 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" event={"ID":"f9551403-60a7-4a70-a8e5-aa86043b3185","Type":"ContainerStarted","Data":"2aa7a709f6e3debddbaa017c2d73f1aa501556704bfffa70480648bef08ad796"} Oct 30 22:07:20.472809 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:20.472755 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" podStartSLOduration=2.472740351 podStartE2EDuration="2.472740351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:20.470673065 +0000 UTC m=+3030.604046309" watchObservedRunningTime="2026-10-30 22:07:20.472740351 +0000 UTC m=+3030.606113545" Oct 30 22:07:20.740730 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:20.740635 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qxjl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 22:07:20.741023 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:20.740980 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 22:07:20.986793 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:20.986765 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:20.988428 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:20.988404 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-f7ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Oct 30 22:07:20.988662 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:20.988644 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Oct 30 22:07:21.736513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:21.734909 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qxjl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 22:07:21.736513 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:21.734972 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 22:07:21.987022 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:21.986942 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-f7ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Oct 30 22:07:21.987022 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:21.986991 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Oct 30 22:07:22.737386 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:22.736927 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qxjl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 22:07:22.737386 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:22.736991 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 22:07:22.986732 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:22.986690 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-f7ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Oct 30 22:07:22.986920 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:22.986749 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Oct 30 22:07:23.740517 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:23.739948 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qxjl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 22:07:23.740517 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:23.740011 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 22:07:23.987343 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:23.987308 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-f7ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Oct 30 22:07:23.987844 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:23.987819 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Oct 30 22:07:24.734916 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:24.734878 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qxjl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 22:07:24.735218 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:24.735190 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 22:07:24.987316 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:24.987237 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-f7ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Oct 30 22:07:24.987316 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:24.987293 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Oct 30 22:07:25.735085 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:25.735047 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qxjl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 22:07:25.735280 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:25.735116 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 22:07:25.986470 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:25.986388 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-f7ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body= Oct 30 22:07:25.986470 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:25.986440 4526 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" Oct 30 22:07:27.519412 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:27.519370 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:27.520255 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:27.520160 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:27.521124 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:27.521099 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:07:27.710044 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:27.710003 4526 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:27.710567 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:27.710504 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:27.711556 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:27.711535 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:07:37.873835 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.873792 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn"] Oct 30 22:07:37.877575 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.877558 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:37.917908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.917871 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97zq\" (UniqueName: \"kubernetes.io/projected/09a63b28-5721-4031-8122-b8ef2c0c2f46-kube-api-access-m97zq\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:37.918091 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.917930 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:37.918091 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.918031 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:37.930087 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.930063 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:37.978926 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.978886 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:37.979138 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.979024 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:37.979138 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.979050 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:07:37.984184 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:37.984161 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn"] Oct 30 22:07:38.018703 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:38.018670 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:38.018900 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:38.018722 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:38.018900 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:38.018783 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97zq\" (UniqueName: \"kubernetes.io/projected/09a63b28-5721-4031-8122-b8ef2c0c2f46-kube-api-access-m97zq\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:38.019456 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:38.019435 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:38.021289 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:38.021273 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:38.077452 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:38.077421 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97zq\" (UniqueName: \"kubernetes.io/projected/09a63b28-5721-4031-8122-b8ef2c0c2f46-kube-api-access-m97zq\") pod \"maas-api-revoke-sub-2e1f5b5f-6frxn\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:38.191325 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:38.191231 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:38.567345 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:38.567323 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn"] Oct 30 22:07:38.569870 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:07:38.569845 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a63b28_5721_4031_8122_b8ef2c0c2f46.slice/crio-7248eefa7ecef1704f7f46ca12eb233149e17cf5951c24ccfdb3addef3256c0a WatchSource:0}: Error finding container 7248eefa7ecef1704f7f46ca12eb233149e17cf5951c24ccfdb3addef3256c0a: Status 404 returned error can't find the container with id 7248eefa7ecef1704f7f46ca12eb233149e17cf5951c24ccfdb3addef3256c0a Oct 30 22:07:39.496192 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:39.496154 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" event={"ID":"09a63b28-5721-4031-8122-b8ef2c0c2f46","Type":"ContainerStarted","Data":"1b7247626bc8d965390bdefcc94efb6ca93551f496f1604a69c5d05a25821c06"} Oct 30 22:07:39.496605 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:39.496217 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" event={"ID":"09a63b28-5721-4031-8122-b8ef2c0c2f46","Type":"ContainerStarted","Data":"7248eefa7ecef1704f7f46ca12eb233149e17cf5951c24ccfdb3addef3256c0a"} Oct 30 22:07:39.609930 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:39.609876 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" podStartSLOduration=2.609862031 podStartE2EDuration="2.609862031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:39.608308309 +0000 UTC m=+3049.741681507" watchObservedRunningTime="2026-10-30 22:07:39.609862031 +0000 UTC m=+3049.743235229" Oct 30 22:07:40.503141 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:40.503104 4526 generic.go:356] "Generic (PLEG): container finished" podID="09a63b28-5721-4031-8122-b8ef2c0c2f46" containerID="1b7247626bc8d965390bdefcc94efb6ca93551f496f1604a69c5d05a25821c06" exitCode=0 Oct 30 22:07:40.503599 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:40.503154 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" event={"ID":"09a63b28-5721-4031-8122-b8ef2c0c2f46","Type":"ContainerDied","Data":"1b7247626bc8d965390bdefcc94efb6ca93551f496f1604a69c5d05a25821c06"} Oct 30 22:07:41.650403 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.650373 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:41.752437 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.752396 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a63b28-5721-4031-8122-b8ef2c0c2f46-kube-api-access-m97zq\" (UniqueName: \"kubernetes.io/projected/09a63b28-5721-4031-8122-b8ef2c0c2f46-kube-api-access-m97zq\") pod \"09a63b28-5721-4031-8122-b8ef2c0c2f46\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " Oct 30 22:07:41.752664 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.752568 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-openshift-ca\") pod \"09a63b28-5721-4031-8122-b8ef2c0c2f46\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " Oct 30 22:07:41.752664 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.752628 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-xks-ca\") pod \"09a63b28-5721-4031-8122-b8ef2c0c2f46\" (UID: \"09a63b28-5721-4031-8122-b8ef2c0c2f46\") " Oct 30 22:07:41.752952 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.752925 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-openshift-ca" pod "09a63b28-5721-4031-8122-b8ef2c0c2f46" (UID: "09a63b28-5721-4031-8122-b8ef2c0c2f46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:41.754694 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.754672 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-xks-ca" pod "09a63b28-5721-4031-8122-b8ef2c0c2f46" (UID: "09a63b28-5721-4031-8122-b8ef2c0c2f46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:41.754758 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.754722 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a63b28-5721-4031-8122-b8ef2c0c2f46-kube-api-access-m97zq" pod "09a63b28-5721-4031-8122-b8ef2c0c2f46" (UID: "09a63b28-5721-4031-8122-b8ef2c0c2f46"). InnerVolumeSpecName "kube-api-access-m97zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:41.853871 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.853830 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:41.853871 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.853865 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09a63b28-5721-4031-8122-b8ef2c0c2f46-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:41.853871 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:41.853875 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m97zq\" (UniqueName: \"kubernetes.io/projected/09a63b28-5721-4031-8122-b8ef2c0c2f46-kube-api-access-m97zq\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:42.477432 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:07:42.477393 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:07:42.514150 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:42.514125 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" Oct 30 22:07:42.514322 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:42.514165 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn" event={"ID":"09a63b28-5721-4031-8122-b8ef2c0c2f46","Type":"ContainerDied","Data":"7248eefa7ecef1704f7f46ca12eb233149e17cf5951c24ccfdb3addef3256c0a"} Oct 30 22:07:42.514322 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:42.514211 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7248eefa7ecef1704f7f46ca12eb233149e17cf5951c24ccfdb3addef3256c0a" Oct 30 22:07:50.394567 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.394535 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh"] Oct 30 22:07:50.394935 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.394885 4526 cpu_manager.go:414] "containerMap: removing container" podUID="09a63b28-5721-4031-8122-b8ef2c0c2f46" containerName="revoke-keys" Oct 30 22:07:50.394935 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.394896 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a63b28-5721-4031-8122-b8ef2c0c2f46" containerName="revoke-keys" Oct 30 22:07:50.395015 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.394961 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a63b28-5721-4031-8122-b8ef2c0c2f46" containerName="revoke-keys" Oct 30 22:07:50.400461 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.400444 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.456985 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.456957 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh"] Oct 30 22:07:50.503646 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.503615 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:50.503818 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.503671 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:50.503818 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.503731 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:50.503818 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.503782 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:07:50.535814 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.535788 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.535923 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.535833 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.535923 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.535857 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvzp\" (UniqueName: \"kubernetes.io/projected/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-kube-api-access-9rvzp\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.637261 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.637231 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.637382 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.637276 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.637382 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.637299 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvzp\" (UniqueName: \"kubernetes.io/projected/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-kube-api-access-9rvzp\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.637911 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.637888 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.639793 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.639775 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.697897 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.697845 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvzp\" (UniqueName: \"kubernetes.io/projected/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-kube-api-access-9rvzp\") pod \"maas-api-revoke-sub-ec843018-dpzxh\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:50.713138 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:50.713119 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:51.085840 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:51.085801 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh"] Oct 30 22:07:51.088143 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:07:51.088113 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcf6f6a_87ef_4ec7_a3f9_8130d1295a46.slice/crio-b84600e476c42a79bae18b9f58d5ef0098d4f1e27fbc2be39d79350210dfc749 WatchSource:0}: Error finding container b84600e476c42a79bae18b9f58d5ef0098d4f1e27fbc2be39d79350210dfc749: Status 404 returned error can't find the container with id b84600e476c42a79bae18b9f58d5ef0098d4f1e27fbc2be39d79350210dfc749 Oct 30 22:07:51.561040 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:51.561005 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" event={"ID":"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46","Type":"ContainerStarted","Data":"dcbbe767e3f1977bea89226851b46fe9578e66b1a7ab0045daa761b3db4ae280"} Oct 30 22:07:51.561469 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:51.561058 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" event={"ID":"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46","Type":"ContainerStarted","Data":"b84600e476c42a79bae18b9f58d5ef0098d4f1e27fbc2be39d79350210dfc749"} Oct 30 22:07:51.675674 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:51.675611 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" podStartSLOduration=1.67559323 podStartE2EDuration="1.67559323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:51.674142198 +0000 UTC m=+3061.807515395" watchObservedRunningTime="2026-10-30 22:07:51.67559323 +0000 UTC m=+3061.808966430" Oct 30 22:07:52.567422 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:52.567392 4526 generic.go:356] "Generic (PLEG): container finished" podID="afcf6f6a-87ef-4ec7-a3f9-8130d1295a46" containerID="dcbbe767e3f1977bea89226851b46fe9578e66b1a7ab0045daa761b3db4ae280" exitCode=0 Oct 30 22:07:52.567871 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:52.567496 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" event={"ID":"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46","Type":"ContainerDied","Data":"dcbbe767e3f1977bea89226851b46fe9578e66b1a7ab0045daa761b3db4ae280"} Oct 30 22:07:52.831768 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:52.831667 4526 scope.go:122] "RemoveContainer" containerID="4458bbdfab40a90c213c66213512b178b9fc7e513cf63ce633a021aa0ed55767" Oct 30 22:07:53.703838 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.703808 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:53.775911 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.775878 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-xks-ca\") pod \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " Oct 30 22:07:53.776116 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.775917 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-kube-api-access-9rvzp\" (UniqueName: \"kubernetes.io/projected/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-kube-api-access-9rvzp\") pod \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " Oct 30 22:07:53.776116 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.775950 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-openshift-ca\") pod \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\" (UID: \"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46\") " Oct 30 22:07:53.776344 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.776320 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-openshift-ca" pod "afcf6f6a-87ef-4ec7-a3f9-8130d1295a46" (UID: "afcf6f6a-87ef-4ec7-a3f9-8130d1295a46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:53.778232 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.778211 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-kube-api-access-9rvzp" pod "afcf6f6a-87ef-4ec7-a3f9-8130d1295a46" (UID: "afcf6f6a-87ef-4ec7-a3f9-8130d1295a46"). InnerVolumeSpecName "kube-api-access-9rvzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:53.778320 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.778278 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-xks-ca" pod "afcf6f6a-87ef-4ec7-a3f9-8130d1295a46" (UID: "afcf6f6a-87ef-4ec7-a3f9-8130d1295a46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:53.876982 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.876910 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:53.876982 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.876937 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvzp\" (UniqueName: \"kubernetes.io/projected/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-kube-api-access-9rvzp\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:53.876982 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:53.876947 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:54.579184 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:54.579143 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" event={"ID":"afcf6f6a-87ef-4ec7-a3f9-8130d1295a46","Type":"ContainerDied","Data":"b84600e476c42a79bae18b9f58d5ef0098d4f1e27fbc2be39d79350210dfc749"} Oct 30 22:07:54.579377 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:54.579201 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh" Oct 30 22:07:54.579377 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:07:54.579225 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b84600e476c42a79bae18b9f58d5ef0098d4f1e27fbc2be39d79350210dfc749" Oct 30 22:08:12.478025 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:08:12.477979 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:08:25.469125 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.469087 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j"] Oct 30 22:08:25.469579 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.469510 4526 cpu_manager.go:414] "containerMap: removing container" podUID="afcf6f6a-87ef-4ec7-a3f9-8130d1295a46" containerName="revoke-keys" Oct 30 22:08:25.469579 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.469523 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcf6f6a-87ef-4ec7-a3f9-8130d1295a46" containerName="revoke-keys" Oct 30 22:08:25.469659 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.469616 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcf6f6a-87ef-4ec7-a3f9-8130d1295a46" containerName="revoke-keys" Oct 30 22:08:25.472357 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.472340 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.524506 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.524460 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:25.552992 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.552955 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:08:25.552992 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.552965 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:25.573794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.573763 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:25.576160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.576130 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j"] Oct 30 22:08:25.601556 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.601517 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.601556 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.601555 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnqql\" (UniqueName: \"kubernetes.io/projected/8a5196f6-c994-41ee-a330-06766bb590ab-kube-api-access-xnqql\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.601788 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.601606 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.702870 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.702829 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.703064 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.702925 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.703064 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.702944 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnqql\" (UniqueName: \"kubernetes.io/projected/8a5196f6-c994-41ee-a330-06766bb590ab-kube-api-access-xnqql\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.703755 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.703733 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.705514 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.705482 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.760978 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.760903 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnqql\" (UniqueName: \"kubernetes.io/projected/8a5196f6-c994-41ee-a330-06766bb590ab-kube-api-access-xnqql\") pod \"maas-api-revoke-sub-046e1343-pxz6j\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:25.785629 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:25.785596 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:26.160068 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:26.160044 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j"] Oct 30 22:08:26.162660 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:08:26.162631 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5196f6_c994_41ee_a330_06766bb590ab.slice/crio-31408e7519862e4da7178201bb37875066d7b53d24d694d6c328b3bde9fa0fac WatchSource:0}: Error finding container 31408e7519862e4da7178201bb37875066d7b53d24d694d6c328b3bde9fa0fac: Status 404 returned error can't find the container with id 31408e7519862e4da7178201bb37875066d7b53d24d694d6c328b3bde9fa0fac Oct 30 22:08:26.757247 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:26.757209 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" event={"ID":"8a5196f6-c994-41ee-a330-06766bb590ab","Type":"ContainerStarted","Data":"75f482522186cefad74c5f898ca68e998669485d2006d8c0f5dfd9fdf3ddee5a"} Oct 30 22:08:26.757699 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:26.757266 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" event={"ID":"8a5196f6-c994-41ee-a330-06766bb590ab","Type":"ContainerStarted","Data":"31408e7519862e4da7178201bb37875066d7b53d24d694d6c328b3bde9fa0fac"} Oct 30 22:08:26.870742 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:26.870687 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" podStartSLOduration=1.870671626 podStartE2EDuration="1.870671626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:26.868783306 +0000 UTC m=+3097.002156507" watchObservedRunningTime="2026-10-30 22:08:26.870671626 +0000 UTC m=+3097.004044824" Oct 30 22:08:27.763571 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:27.763537 4526 generic.go:356] "Generic (PLEG): container finished" podID="8a5196f6-c994-41ee-a330-06766bb590ab" containerID="75f482522186cefad74c5f898ca68e998669485d2006d8c0f5dfd9fdf3ddee5a" exitCode=0 Oct 30 22:08:27.763986 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:27.763616 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" event={"ID":"8a5196f6-c994-41ee-a330-06766bb590ab","Type":"ContainerDied","Data":"75f482522186cefad74c5f898ca68e998669485d2006d8c0f5dfd9fdf3ddee5a"} Oct 30 22:08:28.910655 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:28.910632 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:29.034296 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.034273 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-openshift-ca\") pod \"8a5196f6-c994-41ee-a330-06766bb590ab\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " Oct 30 22:08:29.034618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.034337 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5196f6-c994-41ee-a330-06766bb590ab-kube-api-access-xnqql\" (UniqueName: \"kubernetes.io/projected/8a5196f6-c994-41ee-a330-06766bb590ab-kube-api-access-xnqql\") pod \"8a5196f6-c994-41ee-a330-06766bb590ab\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " Oct 30 22:08:29.034618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.034367 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-xks-ca\") pod \"8a5196f6-c994-41ee-a330-06766bb590ab\" (UID: \"8a5196f6-c994-41ee-a330-06766bb590ab\") " Oct 30 22:08:29.034758 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.034686 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-openshift-ca" pod "8a5196f6-c994-41ee-a330-06766bb590ab" (UID: "8a5196f6-c994-41ee-a330-06766bb590ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:29.036591 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.036562 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5196f6-c994-41ee-a330-06766bb590ab-kube-api-access-xnqql" pod "8a5196f6-c994-41ee-a330-06766bb590ab" (UID: "8a5196f6-c994-41ee-a330-06766bb590ab"). InnerVolumeSpecName "kube-api-access-xnqql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:29.036806 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.036783 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-xks-ca" pod "8a5196f6-c994-41ee-a330-06766bb590ab" (UID: "8a5196f6-c994-41ee-a330-06766bb590ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:29.136069 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.136032 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:29.136069 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.136062 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnqql\" (UniqueName: \"kubernetes.io/projected/8a5196f6-c994-41ee-a330-06766bb590ab-kube-api-access-xnqql\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:29.136069 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.136073 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5196f6-c994-41ee-a330-06766bb590ab-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:29.777458 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.777418 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" Oct 30 22:08:29.777458 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.777440 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j" event={"ID":"8a5196f6-c994-41ee-a330-06766bb590ab","Type":"ContainerDied","Data":"31408e7519862e4da7178201bb37875066d7b53d24d694d6c328b3bde9fa0fac"} Oct 30 22:08:29.777703 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:29.777520 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31408e7519862e4da7178201bb37875066d7b53d24d694d6c328b3bde9fa0fac" Oct 30 22:08:36.272933 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.272849 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc"] Oct 30 22:08:36.273400 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.273379 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8a5196f6-c994-41ee-a330-06766bb590ab" containerName="revoke-keys" Oct 30 22:08:36.273480 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.273404 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5196f6-c994-41ee-a330-06766bb590ab" containerName="revoke-keys" Oct 30 22:08:36.273576 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.273558 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5196f6-c994-41ee-a330-06766bb590ab" containerName="revoke-keys" Oct 30 22:08:36.276878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.276857 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.333897 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.333845 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc"] Oct 30 22:08:36.380159 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.380120 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:36.380159 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.380121 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:36.380159 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.380128 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:08:36.380400 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.380188 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:36.406659 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.406622 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.406833 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.406684 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.406877 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.406822 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgr5x\" (UniqueName: \"kubernetes.io/projected/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-kube-api-access-jgr5x\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.507890 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.507851 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.508110 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.507911 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.508110 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.507968 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgr5x\" (UniqueName: \"kubernetes.io/projected/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-kube-api-access-jgr5x\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.508646 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.508617 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.510439 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.510414 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.564591 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.564562 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgr5x\" (UniqueName: \"kubernetes.io/projected/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-kube-api-access-jgr5x\") pod \"maas-api-revoke-sub-d3fba056-hx8cc\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.589259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.589226 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:36.965381 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:36.965352 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc"] Oct 30 22:08:36.967170 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:08:36.967140 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6ac7b7_989e_4bfc_bfb8_b3b467232fe6.slice/crio-75f9805eadcd98f2f6c445637a96a2f1beef49d503fcd3366c84759c5e190086 WatchSource:0}: Error finding container 75f9805eadcd98f2f6c445637a96a2f1beef49d503fcd3366c84759c5e190086: Status 404 returned error can't find the container with id 75f9805eadcd98f2f6c445637a96a2f1beef49d503fcd3366c84759c5e190086 Oct 30 22:08:37.820653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:37.820611 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" event={"ID":"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6","Type":"ContainerStarted","Data":"51039831c24e835612183e90998b03f82c045f97e30b6d7b8d974ee2c5af0d2a"} Oct 30 22:08:37.821062 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:37.820679 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" event={"ID":"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6","Type":"ContainerStarted","Data":"75f9805eadcd98f2f6c445637a96a2f1beef49d503fcd3366c84759c5e190086"} Oct 30 22:08:37.934475 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:37.934381 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" podStartSLOduration=1.93436817 podStartE2EDuration="1.93436817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:37.934023212 +0000 UTC m=+3108.067396411" watchObservedRunningTime="2026-10-30 22:08:37.93436817 +0000 UTC m=+3108.067741367" Oct 30 22:08:38.830343 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:38.830298 4526 generic.go:356] "Generic (PLEG): container finished" podID="fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6" containerID="51039831c24e835612183e90998b03f82c045f97e30b6d7b8d974ee2c5af0d2a" exitCode=0 Oct 30 22:08:38.830775 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:38.830378 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" event={"ID":"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6","Type":"ContainerDied","Data":"51039831c24e835612183e90998b03f82c045f97e30b6d7b8d974ee2c5af0d2a"} Oct 30 22:08:40.190909 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.190886 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:40.377518 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.363691 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-kube-api-access-jgr5x\" (UniqueName: \"kubernetes.io/projected/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-kube-api-access-jgr5x\") pod \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " Oct 30 22:08:40.377518 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.363788 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-openshift-ca\") pod \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " Oct 30 22:08:40.377518 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.363836 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-xks-ca\") pod \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\" (UID: \"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6\") " Oct 30 22:08:40.378878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.378812 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-xks-ca" pod "fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6" (UID: "fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:40.379180 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.379129 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-openshift-ca" pod "fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6" (UID: "fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:40.380404 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.380376 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-kube-api-access-jgr5x" pod "fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6" (UID: "fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6"). InnerVolumeSpecName "kube-api-access-jgr5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:40.465401 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.465235 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgr5x\" (UniqueName: \"kubernetes.io/projected/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-kube-api-access-jgr5x\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:40.465401 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.465270 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:40.465401 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.465280 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:40.869218 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.868958 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" Oct 30 22:08:40.869218 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.869007 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc" event={"ID":"fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6","Type":"ContainerDied","Data":"75f9805eadcd98f2f6c445637a96a2f1beef49d503fcd3366c84759c5e190086"} Oct 30 22:08:40.869218 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:40.869052 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f9805eadcd98f2f6c445637a96a2f1beef49d503fcd3366c84759c5e190086" Oct 30 22:08:42.477873 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:08:42.477834 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:08:52.862359 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:52.862331 4526 scope.go:122] "RemoveContainer" containerID="9ffdc701bcc424f0d0b940cd7a3ab40c785909d1d4c0abf912bb2008b0b86da8" Oct 30 22:08:52.873625 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:52.873603 4526 scope.go:122] "RemoveContainer" containerID="9981fc1eb9e49b53964304fc6968d8d85ce350fb9835e665cbf9686bcaa311bf" Oct 30 22:08:52.881820 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:08:52.881798 4526 scope.go:122] "RemoveContainer" containerID="e80327752dfdfaa73033f622d445104ad5d02a3df55da025ebb936a63c3ea4b3" Oct 30 22:09:11.116939 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.116896 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422"] Oct 30 22:09:11.117595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.117328 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6" containerName="revoke-keys" Oct 30 22:09:11.117595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.117340 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6" containerName="revoke-keys" Oct 30 22:09:11.117595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.117413 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6" containerName="revoke-keys" Oct 30 22:09:11.122332 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.122309 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.194916 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.194878 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsn2w\" (UniqueName: \"kubernetes.io/projected/c45bb786-2602-4b18-8e7a-6c19d9978ba0-kube-api-access-wsn2w\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.195099 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.195026 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.195149 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.195108 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.224390 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.224353 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:11.224390 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.224373 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:11.224650 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.224372 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:11.224650 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.224411 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:09:11.226646 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.226625 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422"] Oct 30 22:09:11.296391 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.296343 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.296628 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.296418 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.296628 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.296442 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsn2w\" (UniqueName: \"kubernetes.io/projected/c45bb786-2602-4b18-8e7a-6c19d9978ba0-kube-api-access-wsn2w\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.297042 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.297023 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.298922 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.298905 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.353454 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.353417 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsn2w\" (UniqueName: \"kubernetes.io/projected/c45bb786-2602-4b18-8e7a-6c19d9978ba0-kube-api-access-wsn2w\") pod \"maas-api-revoke-sub-ae594979-nx422\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.435396 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.435295 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:11.813806 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:11.813782 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422"] Oct 30 22:09:11.816116 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:09:11.816087 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45bb786_2602_4b18_8e7a_6c19d9978ba0.slice/crio-fa9837c003c1040629caade3590046b25681cc0f581db78a787217c7f6d7d212 WatchSource:0}: Error finding container fa9837c003c1040629caade3590046b25681cc0f581db78a787217c7f6d7d212: Status 404 returned error can't find the container with id fa9837c003c1040629caade3590046b25681cc0f581db78a787217c7f6d7d212 Oct 30 22:09:12.041916 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:12.041881 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" event={"ID":"c45bb786-2602-4b18-8e7a-6c19d9978ba0","Type":"ContainerStarted","Data":"fa9837c003c1040629caade3590046b25681cc0f581db78a787217c7f6d7d212"} Oct 30 22:09:12.477450 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:09:12.477364 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:09:13.048541 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:13.048507 4526 generic.go:356] "Generic (PLEG): container finished" podID="c45bb786-2602-4b18-8e7a-6c19d9978ba0" containerID="d09614d084cb958b1e525c34e489fec8e04022c66b1385f5dc3440c357b66a69" exitCode=0 Oct 30 22:09:13.048740 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:13.048590 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" event={"ID":"c45bb786-2602-4b18-8e7a-6c19d9978ba0","Type":"ContainerDied","Data":"d09614d084cb958b1e525c34e489fec8e04022c66b1385f5dc3440c357b66a69"} Oct 30 22:09:14.189206 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.189182 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:14.328331 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.328242 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-openshift-ca\") pod \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " Oct 30 22:09:14.328507 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.328353 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45bb786-2602-4b18-8e7a-6c19d9978ba0-kube-api-access-wsn2w\" (UniqueName: \"kubernetes.io/projected/c45bb786-2602-4b18-8e7a-6c19d9978ba0-kube-api-access-wsn2w\") pod \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " Oct 30 22:09:14.328507 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.328405 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-xks-ca\") pod \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\" (UID: \"c45bb786-2602-4b18-8e7a-6c19d9978ba0\") " Oct 30 22:09:14.328723 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.328699 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-openshift-ca" pod "c45bb786-2602-4b18-8e7a-6c19d9978ba0" (UID: "c45bb786-2602-4b18-8e7a-6c19d9978ba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:14.330662 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.330627 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-xks-ca" pod "c45bb786-2602-4b18-8e7a-6c19d9978ba0" (UID: "c45bb786-2602-4b18-8e7a-6c19d9978ba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:14.330827 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.330804 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45bb786-2602-4b18-8e7a-6c19d9978ba0-kube-api-access-wsn2w" pod "c45bb786-2602-4b18-8e7a-6c19d9978ba0" (UID: "c45bb786-2602-4b18-8e7a-6c19d9978ba0"). InnerVolumeSpecName "kube-api-access-wsn2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:14.429885 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.429843 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsn2w\" (UniqueName: \"kubernetes.io/projected/c45bb786-2602-4b18-8e7a-6c19d9978ba0-kube-api-access-wsn2w\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.429885 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.429878 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:14.429885 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:14.429894 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45bb786-2602-4b18-8e7a-6c19d9978ba0-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:15.059078 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:15.059049 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" Oct 30 22:09:15.059267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:15.059079 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422" event={"ID":"c45bb786-2602-4b18-8e7a-6c19d9978ba0","Type":"ContainerDied","Data":"fa9837c003c1040629caade3590046b25681cc0f581db78a787217c7f6d7d212"} Oct 30 22:09:15.059267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:15.059125 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa9837c003c1040629caade3590046b25681cc0f581db78a787217c7f6d7d212" Oct 30 22:09:29.048605 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:29.048567 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7"] Oct 30 22:09:29.101043 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:29.101006 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw"] Oct 30 22:09:29.103791 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:29.103765 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b1c9f9c-b5qj7"] Oct 30 22:09:29.107008 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:29.106973 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ee99f3f-s9jbw"] Oct 30 22:09:29.984073 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:29.984030 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e29896-28f5-41e1-9337-e51dc19299b4" path="/var/lib/kubelet/pods/72e29896-28f5-41e1-9337-e51dc19299b4/volumes" Oct 30 22:09:29.984582 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:29.984562 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90db392-1a2d-4f24-ab28-6ca7b0274107" path="/var/lib/kubelet/pods/f90db392-1a2d-4f24-ab28-6ca7b0274107/volumes" Oct 30 22:09:34.051391 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:34.051356 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln"] Oct 30 22:09:34.100886 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:34.100855 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl"] Oct 30 22:09:34.102762 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:34.102750 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-772a8b8a-t48ln"] Oct 30 22:09:34.104262 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:34.104240 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a81e38c-xdfjl"] Oct 30 22:09:35.983595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:35.983565 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57944f20-befb-4a33-8288-1aa2b87024de" path="/var/lib/kubelet/pods/57944f20-befb-4a33-8288-1aa2b87024de/volumes" Oct 30 22:09:35.984009 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:35.983995 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1df5bca-fc84-4bec-bd46-cd7b804234de" path="/var/lib/kubelet/pods/b1df5bca-fc84-4bec-bd46-cd7b804234de/volumes" Oct 30 22:09:39.236764 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.236723 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t"] Oct 30 22:09:39.237158 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.237088 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c45bb786-2602-4b18-8e7a-6c19d9978ba0" containerName="revoke-keys" Oct 30 22:09:39.237158 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.237099 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45bb786-2602-4b18-8e7a-6c19d9978ba0" containerName="revoke-keys" Oct 30 22:09:39.237249 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.237173 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45bb786-2602-4b18-8e7a-6c19d9978ba0" containerName="revoke-keys" Oct 30 22:09:39.241720 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.241703 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.292374 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.292343 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htph\" (UniqueName: \"kubernetes.io/projected/856efb22-d65c-4f66-a33e-7ffd3605deed-kube-api-access-6htph\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.292601 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.292392 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.292601 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.292540 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.295421 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.295397 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:39.344776 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.344739 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:39.344776 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.344769 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:39.345028 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.344800 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:09:39.347956 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.347922 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t"] Oct 30 22:09:39.393010 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.392971 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.393228 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.393045 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htph\" (UniqueName: \"kubernetes.io/projected/856efb22-d65c-4f66-a33e-7ffd3605deed-kube-api-access-6htph\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.393228 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.393073 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.393813 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.393783 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.395902 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.395874 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.449928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.449890 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htph\" (UniqueName: \"kubernetes.io/projected/856efb22-d65c-4f66-a33e-7ffd3605deed-kube-api-access-6htph\") pod \"maas-api-revoke-sub-a4aef7d7-94f9t\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.556415 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.556366 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:39.928663 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:39.928633 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t"] Oct 30 22:09:39.931466 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:09:39.931433 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856efb22_d65c_4f66_a33e_7ffd3605deed.slice/crio-ef895d361eb2e7cbfef9d88fab590c1575c060648813bd90eb4f44d5746b187b WatchSource:0}: Error finding container ef895d361eb2e7cbfef9d88fab590c1575c060648813bd90eb4f44d5746b187b: Status 404 returned error can't find the container with id ef895d361eb2e7cbfef9d88fab590c1575c060648813bd90eb4f44d5746b187b Oct 30 22:09:40.211368 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:40.211339 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" event={"ID":"856efb22-d65c-4f66-a33e-7ffd3605deed","Type":"ContainerStarted","Data":"ef895d361eb2e7cbfef9d88fab590c1575c060648813bd90eb4f44d5746b187b"} Oct 30 22:09:41.218807 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:41.218772 4526 generic.go:356] "Generic (PLEG): container finished" podID="856efb22-d65c-4f66-a33e-7ffd3605deed" containerID="21cb6ae2cb3db24d3b31595bc37b3ea3f850a9ed05e8d94810ad1624b634641c" exitCode=0 Oct 30 22:09:41.219256 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:41.218856 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" event={"ID":"856efb22-d65c-4f66-a33e-7ffd3605deed","Type":"ContainerDied","Data":"21cb6ae2cb3db24d3b31595bc37b3ea3f850a9ed05e8d94810ad1624b634641c"} Oct 30 22:09:42.358619 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.358593 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:42.417451 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.417423 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856efb22-d65c-4f66-a33e-7ffd3605deed-kube-api-access-6htph\" (UniqueName: \"kubernetes.io/projected/856efb22-d65c-4f66-a33e-7ffd3605deed-kube-api-access-6htph\") pod \"856efb22-d65c-4f66-a33e-7ffd3605deed\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " Oct 30 22:09:42.417697 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.417470 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-xks-ca\") pod \"856efb22-d65c-4f66-a33e-7ffd3605deed\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " Oct 30 22:09:42.417697 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.417560 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-openshift-ca\") pod \"856efb22-d65c-4f66-a33e-7ffd3605deed\" (UID: \"856efb22-d65c-4f66-a33e-7ffd3605deed\") " Oct 30 22:09:42.417915 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.417893 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-openshift-ca" pod "856efb22-d65c-4f66-a33e-7ffd3605deed" (UID: "856efb22-d65c-4f66-a33e-7ffd3605deed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:42.419835 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.419806 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-xks-ca" pod "856efb22-d65c-4f66-a33e-7ffd3605deed" (UID: "856efb22-d65c-4f66-a33e-7ffd3605deed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:42.419951 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.419858 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856efb22-d65c-4f66-a33e-7ffd3605deed-kube-api-access-6htph" pod "856efb22-d65c-4f66-a33e-7ffd3605deed" (UID: "856efb22-d65c-4f66-a33e-7ffd3605deed"). InnerVolumeSpecName "kube-api-access-6htph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:42.479616 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:09:42.478946 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:09:42.518293 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.518254 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:42.518293 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.518286 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6htph\" (UniqueName: \"kubernetes.io/projected/856efb22-d65c-4f66-a33e-7ffd3605deed-kube-api-access-6htph\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:42.518293 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:42.518296 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856efb22-d65c-4f66-a33e-7ffd3605deed-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:43.230006 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:43.229978 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" Oct 30 22:09:43.230006 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:43.229982 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t" event={"ID":"856efb22-d65c-4f66-a33e-7ffd3605deed","Type":"ContainerDied","Data":"ef895d361eb2e7cbfef9d88fab590c1575c060648813bd90eb4f44d5746b187b"} Oct 30 22:09:43.230236 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:43.230038 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef895d361eb2e7cbfef9d88fab590c1575c060648813bd90eb4f44d5746b187b" Oct 30 22:09:52.906965 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:09:52.906934 4526 scope.go:122] "RemoveContainer" containerID="da651ef9d1f7d6aea1b97461230e298aa924d935bd801701ffa2a39ee3c8ad32" Oct 30 22:10:12.476713 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:10:12.476626 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:10:23.840246 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.840209 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6"] Oct 30 22:10:23.840805 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.840603 4526 cpu_manager.go:414] "containerMap: removing container" podUID="856efb22-d65c-4f66-a33e-7ffd3605deed" containerName="revoke-keys" Oct 30 22:10:23.840805 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.840614 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856efb22-d65c-4f66-a33e-7ffd3605deed" containerName="revoke-keys" Oct 30 22:10:23.840805 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.840701 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="856efb22-d65c-4f66-a33e-7ffd3605deed" containerName="revoke-keys" Oct 30 22:10:23.843466 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.843439 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:23.947391 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.947356 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:23.947391 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.947386 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:23.947391 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.947401 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:23.947691 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.947467 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:10:23.949740 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:23.949720 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6"] Oct 30 22:10:24.017384 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.017349 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.017606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.017404 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4c9g\" (UniqueName: \"kubernetes.io/projected/a76fbb69-8ad2-4d44-adc5-bbde503edc46-kube-api-access-r4c9g\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.017606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.017584 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.118970 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.118880 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.118970 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.118971 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.119173 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.119000 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4c9g\" (UniqueName: \"kubernetes.io/projected/a76fbb69-8ad2-4d44-adc5-bbde503edc46-kube-api-access-r4c9g\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.119700 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.119675 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.121461 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.121444 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.175253 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.175223 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4c9g\" (UniqueName: \"kubernetes.io/projected/a76fbb69-8ad2-4d44-adc5-bbde503edc46-kube-api-access-r4c9g\") pod \"maas-api-revoke-sub-79805ca9-rxpb6\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.458831 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.458752 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:24.829312 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:24.829266 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6"] Oct 30 22:10:24.831726 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:10:24.831699 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76fbb69_8ad2_4d44_adc5_bbde503edc46.slice/crio-d4d3fedec9729b26d8e031a0834e66cd1dc8dbf765414c756b5c9158712ec016 WatchSource:0}: Error finding container d4d3fedec9729b26d8e031a0834e66cd1dc8dbf765414c756b5c9158712ec016: Status 404 returned error can't find the container with id d4d3fedec9729b26d8e031a0834e66cd1dc8dbf765414c756b5c9158712ec016 Oct 30 22:10:25.465178 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:25.465104 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" event={"ID":"a76fbb69-8ad2-4d44-adc5-bbde503edc46","Type":"ContainerStarted","Data":"721ac24cb1003d3548e484b970d6fdb5a44b4fd80011abdfdbe54d575ee8663d"} Oct 30 22:10:25.465178 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:25.465162 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" event={"ID":"a76fbb69-8ad2-4d44-adc5-bbde503edc46","Type":"ContainerStarted","Data":"d4d3fedec9729b26d8e031a0834e66cd1dc8dbf765414c756b5c9158712ec016"} Oct 30 22:10:25.578461 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:25.578412 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" podStartSLOduration=2.578395015 podStartE2EDuration="2.578395015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:25.577889609 +0000 UTC m=+3215.711262808" watchObservedRunningTime="2026-10-30 22:10:25.578395015 +0000 UTC m=+3215.711768214" Oct 30 22:10:26.471521 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:26.471467 4526 generic.go:356] "Generic (PLEG): container finished" podID="a76fbb69-8ad2-4d44-adc5-bbde503edc46" containerID="721ac24cb1003d3548e484b970d6fdb5a44b4fd80011abdfdbe54d575ee8663d" exitCode=0 Oct 30 22:10:26.471889 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:26.471555 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" event={"ID":"a76fbb69-8ad2-4d44-adc5-bbde503edc46","Type":"ContainerDied","Data":"721ac24cb1003d3548e484b970d6fdb5a44b4fd80011abdfdbe54d575ee8663d"} Oct 30 22:10:27.612931 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.612908 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:27.651160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.651127 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76fbb69-8ad2-4d44-adc5-bbde503edc46-kube-api-access-r4c9g\" (UniqueName: \"kubernetes.io/projected/a76fbb69-8ad2-4d44-adc5-bbde503edc46-kube-api-access-r4c9g\") pod \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " Oct 30 22:10:27.651314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.651187 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-openshift-ca\") pod \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " Oct 30 22:10:27.651314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.651216 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-xks-ca\") pod \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\" (UID: \"a76fbb69-8ad2-4d44-adc5-bbde503edc46\") " Oct 30 22:10:27.651607 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.651578 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-openshift-ca" pod "a76fbb69-8ad2-4d44-adc5-bbde503edc46" (UID: "a76fbb69-8ad2-4d44-adc5-bbde503edc46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:27.653655 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.653620 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-xks-ca" pod "a76fbb69-8ad2-4d44-adc5-bbde503edc46" (UID: "a76fbb69-8ad2-4d44-adc5-bbde503edc46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:27.653761 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.653696 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76fbb69-8ad2-4d44-adc5-bbde503edc46-kube-api-access-r4c9g" pod "a76fbb69-8ad2-4d44-adc5-bbde503edc46" (UID: "a76fbb69-8ad2-4d44-adc5-bbde503edc46"). InnerVolumeSpecName "kube-api-access-r4c9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:27.752254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.752166 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4c9g\" (UniqueName: \"kubernetes.io/projected/a76fbb69-8ad2-4d44-adc5-bbde503edc46-kube-api-access-r4c9g\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:27.752254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.752206 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:27.752254 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:27.752223 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fbb69-8ad2-4d44-adc5-bbde503edc46-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:28.214201 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:28.214157 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podUID="a4207178a950303c256d4f333088d96f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 22:10:28.483204 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:28.483116 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" Oct 30 22:10:28.483204 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:28.483159 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6" event={"ID":"a76fbb69-8ad2-4d44-adc5-bbde503edc46","Type":"ContainerDied","Data":"d4d3fedec9729b26d8e031a0834e66cd1dc8dbf765414c756b5c9158712ec016"} Oct 30 22:10:28.483407 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:28.483208 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d3fedec9729b26d8e031a0834e66cd1dc8dbf765414c756b5c9158712ec016" Oct 30 22:10:41.193694 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:41.193652 4526 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nfl6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:41.194447 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:41.194408 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nfl6x" podUID="d055332f-4f2b-4d52-842b-163859de6009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:10:42.045988 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:42.045953 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf"] Oct 30 22:10:42.048903 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:42.048877 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-9g6hf"] Oct 30 22:10:42.477634 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:10:42.477549 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:10:43.984134 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:43.984093 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fd3368-cd61-40ed-bbf6-d78956bf8448" path="/var/lib/kubelet/pods/59fd3368-cd61-40ed-bbf6-d78956bf8448/volumes" Oct 30 22:10:48.235557 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.235521 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4"] Oct 30 22:10:48.235956 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.235887 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a76fbb69-8ad2-4d44-adc5-bbde503edc46" containerName="revoke-keys" Oct 30 22:10:48.235956 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.235897 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76fbb69-8ad2-4d44-adc5-bbde503edc46" containerName="revoke-keys" Oct 30 22:10:48.236035 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.235975 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76fbb69-8ad2-4d44-adc5-bbde503edc46" containerName="revoke-keys" Oct 30 22:10:48.239428 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.239413 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.293595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.293562 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:48.293789 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.293712 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:48.342947 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.342914 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:10:48.343122 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.342956 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:48.345259 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.345233 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4"] Oct 30 22:10:48.352557 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.352534 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkq9\" (UniqueName: \"kubernetes.io/projected/c1ba8542-615d-4c84-9f99-7d0416e2f258-kube-api-access-lbkq9\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.352640 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.352624 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.352753 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.352736 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.453346 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.453305 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.453590 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.453357 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.453590 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.453396 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkq9\" (UniqueName: \"kubernetes.io/projected/c1ba8542-615d-4c84-9f99-7d0416e2f258-kube-api-access-lbkq9\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.454135 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.454108 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.455890 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.455867 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.510414 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.510336 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkq9\" (UniqueName: \"kubernetes.io/projected/c1ba8542-615d-4c84-9f99-7d0416e2f258-kube-api-access-lbkq9\") pod \"maas-api-revoke-sub-0b7e8edb-ptkz4\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.551884 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.551852 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:48.923789 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:48.923762 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4"] Oct 30 22:10:48.926616 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:10:48.926583 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ba8542_615d_4c84_9f99_7d0416e2f258.slice/crio-c60f95d18346adc16aa8d1cca05eb883d776c9e9b4251b2801cacd0a413ef0b6 WatchSource:0}: Error finding container c60f95d18346adc16aa8d1cca05eb883d776c9e9b4251b2801cacd0a413ef0b6: Status 404 returned error can't find the container with id c60f95d18346adc16aa8d1cca05eb883d776c9e9b4251b2801cacd0a413ef0b6 Oct 30 22:10:49.611308 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:49.611272 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" event={"ID":"c1ba8542-615d-4c84-9f99-7d0416e2f258","Type":"ContainerStarted","Data":"462914b0592ca47d9212d3bde407fdbe8b0e0af3a4f34c79bd9ae474aadfa052"} Oct 30 22:10:49.611744 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:49.611339 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" event={"ID":"c1ba8542-615d-4c84-9f99-7d0416e2f258","Type":"ContainerStarted","Data":"c60f95d18346adc16aa8d1cca05eb883d776c9e9b4251b2801cacd0a413ef0b6"} Oct 30 22:10:49.725293 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:49.725229 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" podStartSLOduration=1.725212354 podStartE2EDuration="1.725212354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:49.723045529 +0000 UTC m=+3239.856418730" watchObservedRunningTime="2026-10-30 22:10:49.725212354 +0000 UTC m=+3239.858585552" Oct 30 22:10:50.617682 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:50.617646 4526 generic.go:356] "Generic (PLEG): container finished" podID="c1ba8542-615d-4c84-9f99-7d0416e2f258" containerID="462914b0592ca47d9212d3bde407fdbe8b0e0af3a4f34c79bd9ae474aadfa052" exitCode=0 Oct 30 22:10:50.618102 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:50.617735 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" event={"ID":"c1ba8542-615d-4c84-9f99-7d0416e2f258","Type":"ContainerDied","Data":"462914b0592ca47d9212d3bde407fdbe8b0e0af3a4f34c79bd9ae474aadfa052"} Oct 30 22:10:51.767865 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.767838 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:51.886262 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.886185 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ba8542-615d-4c84-9f99-7d0416e2f258-kube-api-access-lbkq9\" (UniqueName: \"kubernetes.io/projected/c1ba8542-615d-4c84-9f99-7d0416e2f258-kube-api-access-lbkq9\") pod \"c1ba8542-615d-4c84-9f99-7d0416e2f258\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " Oct 30 22:10:51.886262 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.886257 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-openshift-ca\") pod \"c1ba8542-615d-4c84-9f99-7d0416e2f258\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " Oct 30 22:10:51.886542 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.886311 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-xks-ca\") pod \"c1ba8542-615d-4c84-9f99-7d0416e2f258\" (UID: \"c1ba8542-615d-4c84-9f99-7d0416e2f258\") " Oct 30 22:10:51.886693 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.886666 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-openshift-ca" pod "c1ba8542-615d-4c84-9f99-7d0416e2f258" (UID: "c1ba8542-615d-4c84-9f99-7d0416e2f258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:51.888641 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.888614 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ba8542-615d-4c84-9f99-7d0416e2f258-kube-api-access-lbkq9" pod "c1ba8542-615d-4c84-9f99-7d0416e2f258" (UID: "c1ba8542-615d-4c84-9f99-7d0416e2f258"). InnerVolumeSpecName "kube-api-access-lbkq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:51.888983 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.888961 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-xks-ca" pod "c1ba8542-615d-4c84-9f99-7d0416e2f258" (UID: "c1ba8542-615d-4c84-9f99-7d0416e2f258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:51.987883 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.987848 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbkq9\" (UniqueName: \"kubernetes.io/projected/c1ba8542-615d-4c84-9f99-7d0416e2f258-kube-api-access-lbkq9\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:51.987883 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.987884 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:51.988092 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:51.987900 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ba8542-615d-4c84-9f99-7d0416e2f258-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:52.631082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.631051 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" Oct 30 22:10:52.631082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.631072 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4" event={"ID":"c1ba8542-615d-4c84-9f99-7d0416e2f258","Type":"ContainerDied","Data":"c60f95d18346adc16aa8d1cca05eb883d776c9e9b4251b2801cacd0a413ef0b6"} Oct 30 22:10:52.631329 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.631120 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c60f95d18346adc16aa8d1cca05eb883d776c9e9b4251b2801cacd0a413ef0b6" Oct 30 22:10:52.927945 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.927863 4526 scope.go:122] "RemoveContainer" containerID="86f05b21391e9fbe86a444bfd8d3cad3ec17f8c9f81aba67212125dafadc66e9" Oct 30 22:10:52.936898 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.936879 4526 scope.go:122] "RemoveContainer" containerID="93460357cc58901f1f36e5900640c4e0b33cbb8a3e440131ce58a3e5701448a6" Oct 30 22:10:52.947650 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.947628 4526 scope.go:122] "RemoveContainer" containerID="1bb78fb0319957136b4994bdaca9edd2c20ab9528f73e71ca3aad8b6c2bd6e7f" Oct 30 22:10:52.955640 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.955621 4526 scope.go:122] "RemoveContainer" containerID="68b2c95e0971d2cac8e256cd256a579fb4b27c7552579673879c03820661087a" Oct 30 22:10:52.963622 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.963605 4526 scope.go:122] "RemoveContainer" containerID="a02a67e9cf8209b4beb9d04b5feaf1df59279f4480814398647694a5fd8a82db" Oct 30 22:10:52.971237 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:52.971212 4526 scope.go:122] "RemoveContainer" containerID="53c0cff847c29f016fae10a3e780daa921cb4af8011c85f34bad5d740e56f40a" Oct 30 22:10:59.635071 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.635033 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr"] Oct 30 22:10:59.635624 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.635466 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c1ba8542-615d-4c84-9f99-7d0416e2f258" containerName="revoke-keys" Oct 30 22:10:59.635624 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.635479 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ba8542-615d-4c84-9f99-7d0416e2f258" containerName="revoke-keys" Oct 30 22:10:59.635624 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.635597 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ba8542-615d-4c84-9f99-7d0416e2f258" containerName="revoke-keys" Oct 30 22:10:59.641930 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.641904 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.696956 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.696924 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr"] Oct 30 22:10:59.743084 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.743048 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:59.743084 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.743048 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:59.743297 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.743065 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:59.744708 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.744689 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:10:59.763537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.763480 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.763705 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.763617 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lth\" (UniqueName: \"kubernetes.io/projected/ec22e93b-2509-4e75-9cf4-0554caf6772d-kube-api-access-s4lth\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.763705 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.763666 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.864899 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.864862 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.865081 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.864919 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lth\" (UniqueName: \"kubernetes.io/projected/ec22e93b-2509-4e75-9cf4-0554caf6772d-kube-api-access-s4lth\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.865081 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.864955 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.865712 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.865686 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.867440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.867417 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.921515 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.921426 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lth\" (UniqueName: \"kubernetes.io/projected/ec22e93b-2509-4e75-9cf4-0554caf6772d-kube-api-access-s4lth\") pod \"maas-api-revoke-sub-3d4156f1-k4mkr\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:10:59.954638 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:10:59.954590 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:11:00.325892 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:00.325868 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr"] Oct 30 22:11:00.328398 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:11:00.328364 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec22e93b_2509_4e75_9cf4_0554caf6772d.slice/crio-fa5cded62ba013cc9597ea243b02db38caf663a477ae01fb0c0c0683ea9e60ce WatchSource:0}: Error finding container fa5cded62ba013cc9597ea243b02db38caf663a477ae01fb0c0c0683ea9e60ce: Status 404 returned error can't find the container with id fa5cded62ba013cc9597ea243b02db38caf663a477ae01fb0c0c0683ea9e60ce Oct 30 22:11:00.673202 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:00.673169 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" event={"ID":"ec22e93b-2509-4e75-9cf4-0554caf6772d","Type":"ContainerStarted","Data":"183a637df12f20d31ede3c76e820d0b5b0e763ddc09c8e5040c263e03aecfbd8"} Oct 30 22:11:00.673721 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:00.673222 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" event={"ID":"ec22e93b-2509-4e75-9cf4-0554caf6772d","Type":"ContainerStarted","Data":"fa5cded62ba013cc9597ea243b02db38caf663a477ae01fb0c0c0683ea9e60ce"} Oct 30 22:11:00.787084 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:00.787029 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" podStartSLOduration=1.787012968 podStartE2EDuration="1.787012968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:00.785559419 +0000 UTC m=+3250.918932617" watchObservedRunningTime="2026-10-30 22:11:00.787012968 +0000 UTC m=+3250.920386165" Oct 30 22:11:01.678819 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:01.678783 4526 generic.go:356] "Generic (PLEG): container finished" podID="ec22e93b-2509-4e75-9cf4-0554caf6772d" containerID="183a637df12f20d31ede3c76e820d0b5b0e763ddc09c8e5040c263e03aecfbd8" exitCode=0 Oct 30 22:11:01.679215 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:01.678829 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" event={"ID":"ec22e93b-2509-4e75-9cf4-0554caf6772d","Type":"ContainerDied","Data":"183a637df12f20d31ede3c76e820d0b5b0e763ddc09c8e5040c263e03aecfbd8"} Oct 30 22:11:02.819874 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.819844 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:11:02.898197 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.898156 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-xks-ca\") pod \"ec22e93b-2509-4e75-9cf4-0554caf6772d\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " Oct 30 22:11:02.898400 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.898268 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec22e93b-2509-4e75-9cf4-0554caf6772d-kube-api-access-s4lth\" (UniqueName: \"kubernetes.io/projected/ec22e93b-2509-4e75-9cf4-0554caf6772d-kube-api-access-s4lth\") pod \"ec22e93b-2509-4e75-9cf4-0554caf6772d\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " Oct 30 22:11:02.898400 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.898301 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-openshift-ca\") pod \"ec22e93b-2509-4e75-9cf4-0554caf6772d\" (UID: \"ec22e93b-2509-4e75-9cf4-0554caf6772d\") " Oct 30 22:11:02.898749 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.898725 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-openshift-ca" pod "ec22e93b-2509-4e75-9cf4-0554caf6772d" (UID: "ec22e93b-2509-4e75-9cf4-0554caf6772d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:02.900583 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.900557 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec22e93b-2509-4e75-9cf4-0554caf6772d-kube-api-access-s4lth" pod "ec22e93b-2509-4e75-9cf4-0554caf6772d" (UID: "ec22e93b-2509-4e75-9cf4-0554caf6772d"). InnerVolumeSpecName "kube-api-access-s4lth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:02.900674 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.900591 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-xks-ca" pod "ec22e93b-2509-4e75-9cf4-0554caf6772d" (UID: "ec22e93b-2509-4e75-9cf4-0554caf6772d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:02.999535 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.999418 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:02.999535 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.999452 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4lth\" (UniqueName: \"kubernetes.io/projected/ec22e93b-2509-4e75-9cf4-0554caf6772d-kube-api-access-s4lth\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:02.999535 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:02.999462 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec22e93b-2509-4e75-9cf4-0554caf6772d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:03.690171 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:03.690139 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" Oct 30 22:11:03.690360 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:03.690142 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr" event={"ID":"ec22e93b-2509-4e75-9cf4-0554caf6772d","Type":"ContainerDied","Data":"fa5cded62ba013cc9597ea243b02db38caf663a477ae01fb0c0c0683ea9e60ce"} Oct 30 22:11:03.690360 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:03.690267 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa5cded62ba013cc9597ea243b02db38caf663a477ae01fb0c0c0683ea9e60ce" Oct 30 22:11:12.476626 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:11:12.476589 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:11:17.643637 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.643599 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr"] Oct 30 22:11:17.693410 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.693375 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k4mkr"] Oct 30 22:11:17.762629 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.762596 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc"] Oct 30 22:11:17.763006 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.762992 4526 cpu_manager.go:414] "containerMap: removing container" podUID="ec22e93b-2509-4e75-9cf4-0554caf6772d" containerName="revoke-keys" Oct 30 22:11:17.763061 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.763008 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec22e93b-2509-4e75-9cf4-0554caf6772d" containerName="revoke-keys" Oct 30 22:11:17.763100 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.763086 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec22e93b-2509-4e75-9cf4-0554caf6772d" containerName="revoke-keys" Oct 30 22:11:17.766463 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.766441 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.818770 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.818737 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:17.818993 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.818857 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:17.818993 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.818890 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:17.818993 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.818936 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:11:17.826561 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.826528 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc"] Oct 30 22:11:17.844973 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.844937 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.845194 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.844986 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.845194 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.845133 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxdh\" (UniqueName: \"kubernetes.io/projected/69a5a772-193d-4599-afe5-0aee07c59935-kube-api-access-kqxdh\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.946239 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.946125 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.946239 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.946193 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.946475 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.946262 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxdh\" (UniqueName: \"kubernetes.io/projected/69a5a772-193d-4599-afe5-0aee07c59935-kube-api-access-kqxdh\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.946944 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.946867 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.948750 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.948724 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:17.983730 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:17.983700 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec22e93b-2509-4e75-9cf4-0554caf6772d" path="/var/lib/kubelet/pods/ec22e93b-2509-4e75-9cf4-0554caf6772d/volumes" Oct 30 22:11:18.003043 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:18.003015 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxdh\" (UniqueName: \"kubernetes.io/projected/69a5a772-193d-4599-afe5-0aee07c59935-kube-api-access-kqxdh\") pod \"maas-api-revoke-sub-3d4156f1-5bvqc\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:18.079844 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:18.079812 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:18.454139 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:18.454098 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc"] Oct 30 22:11:18.456921 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:11:18.456895 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a5a772_193d_4599_afe5_0aee07c59935.slice/crio-afb63d3047b43e4cfae2fed0accdbb73615a24b9b72667381d9d14382aa5c6b4 WatchSource:0}: Error finding container afb63d3047b43e4cfae2fed0accdbb73615a24b9b72667381d9d14382aa5c6b4: Status 404 returned error can't find the container with id afb63d3047b43e4cfae2fed0accdbb73615a24b9b72667381d9d14382aa5c6b4 Oct 30 22:11:18.762912 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:18.762879 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" event={"ID":"69a5a772-193d-4599-afe5-0aee07c59935","Type":"ContainerStarted","Data":"1827c1773dc01c20bb2e25efdd24cab0d09495a1c97ac0f6235588b93ca576b1"} Oct 30 22:11:18.763326 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:18.762928 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" event={"ID":"69a5a772-193d-4599-afe5-0aee07c59935","Type":"ContainerStarted","Data":"afb63d3047b43e4cfae2fed0accdbb73615a24b9b72667381d9d14382aa5c6b4"} Oct 30 22:11:18.876734 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:18.876668 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" podStartSLOduration=1.876650105 podStartE2EDuration="1.876650105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:18.876415784 +0000 UTC m=+3269.009788994" watchObservedRunningTime="2026-10-30 22:11:18.876650105 +0000 UTC m=+3269.010023282" Oct 30 22:11:19.769936 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:19.769897 4526 generic.go:356] "Generic (PLEG): container finished" podID="69a5a772-193d-4599-afe5-0aee07c59935" containerID="1827c1773dc01c20bb2e25efdd24cab0d09495a1c97ac0f6235588b93ca576b1" exitCode=0 Oct 30 22:11:19.770358 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:19.769991 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" event={"ID":"69a5a772-193d-4599-afe5-0aee07c59935","Type":"ContainerDied","Data":"1827c1773dc01c20bb2e25efdd24cab0d09495a1c97ac0f6235588b93ca576b1"} Oct 30 22:11:20.912629 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:20.912604 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:20.975252 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:20.975221 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-openshift-ca\") pod \"69a5a772-193d-4599-afe5-0aee07c59935\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " Oct 30 22:11:20.975442 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:20.975320 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a5a772-193d-4599-afe5-0aee07c59935-kube-api-access-kqxdh\" (UniqueName: \"kubernetes.io/projected/69a5a772-193d-4599-afe5-0aee07c59935-kube-api-access-kqxdh\") pod \"69a5a772-193d-4599-afe5-0aee07c59935\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " Oct 30 22:11:20.975442 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:20.975407 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-xks-ca\") pod \"69a5a772-193d-4599-afe5-0aee07c59935\" (UID: \"69a5a772-193d-4599-afe5-0aee07c59935\") " Oct 30 22:11:20.975697 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:20.975670 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-openshift-ca" pod "69a5a772-193d-4599-afe5-0aee07c59935" (UID: "69a5a772-193d-4599-afe5-0aee07c59935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:20.977625 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:20.977589 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-xks-ca" pod "69a5a772-193d-4599-afe5-0aee07c59935" (UID: "69a5a772-193d-4599-afe5-0aee07c59935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:20.977751 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:20.977720 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a5a772-193d-4599-afe5-0aee07c59935-kube-api-access-kqxdh" pod "69a5a772-193d-4599-afe5-0aee07c59935" (UID: "69a5a772-193d-4599-afe5-0aee07c59935"). InnerVolumeSpecName "kube-api-access-kqxdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:21.076319 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:21.076282 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.076319 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:21.076308 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a5a772-193d-4599-afe5-0aee07c59935-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.076319 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:21.076318 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqxdh\" (UniqueName: \"kubernetes.io/projected/69a5a772-193d-4599-afe5-0aee07c59935-kube-api-access-kqxdh\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:21.781350 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:21.781317 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" Oct 30 22:11:21.781566 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:21.781351 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc" event={"ID":"69a5a772-193d-4599-afe5-0aee07c59935","Type":"ContainerDied","Data":"afb63d3047b43e4cfae2fed0accdbb73615a24b9b72667381d9d14382aa5c6b4"} Oct 30 22:11:21.781566 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:21.781404 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afb63d3047b43e4cfae2fed0accdbb73615a24b9b72667381d9d14382aa5c6b4" Oct 30 22:11:42.476950 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:11:42.476869 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:11:53.007155 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:11:53.007114 4526 scope.go:122] "RemoveContainer" containerID="fb168ba94119cd3052e9400a92f645e4df864b7f753c5a8145fffb917a5561b8" Oct 30 22:12:07.051090 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:07.051057 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs"] Oct 30 22:12:07.100662 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:07.100623 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dtpgs"] Oct 30 22:12:07.983499 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:07.983458 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7d9181-6a3e-414a-b580-e04f2d713fe6" path="/var/lib/kubelet/pods/dd7d9181-6a3e-414a-b580-e04f2d713fe6/volumes" Oct 30 22:12:12.477845 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:12:12.477801 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:12:15.636657 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.636612 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q"] Oct 30 22:12:15.637083 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.636989 4526 cpu_manager.go:414] "containerMap: removing container" podUID="69a5a772-193d-4599-afe5-0aee07c59935" containerName="revoke-keys" Oct 30 22:12:15.637083 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.637000 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a5a772-193d-4599-afe5-0aee07c59935" containerName="revoke-keys" Oct 30 22:12:15.637083 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.637077 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a5a772-193d-4599-afe5-0aee07c59935" containerName="revoke-keys" Oct 30 22:12:15.641745 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.641727 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.697229 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.697197 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q"] Oct 30 22:12:15.743872 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.743831 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:15.743872 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.743831 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:15.744093 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.743886 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:15.744093 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.743988 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:12:15.746737 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.746711 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrlc\" (UniqueName: \"kubernetes.io/projected/93fccba4-3698-413d-bd1a-11836e94bb20-kube-api-access-fzrlc\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.746849 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.746758 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.746911 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.746893 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.847546 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.847481 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.847757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.847611 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrlc\" (UniqueName: \"kubernetes.io/projected/93fccba4-3698-413d-bd1a-11836e94bb20-kube-api-access-fzrlc\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.847757 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.847637 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.848190 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.848163 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.850182 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.850158 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.904766 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.904688 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrlc\" (UniqueName: \"kubernetes.io/projected/93fccba4-3698-413d-bd1a-11836e94bb20-kube-api-access-fzrlc\") pod \"maas-api-revoke-sub-e2fc9c5b-ccf7q\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:15.957232 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:15.957190 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:16.335020 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:16.334990 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q"] Oct 30 22:12:16.337917 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:12:16.337885 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fccba4_3698_413d_bd1a_11836e94bb20.slice/crio-68c2992804727179410d8d7dd6094308aeb7753d01b7ce304012e7f83d8d6ea5 WatchSource:0}: Error finding container 68c2992804727179410d8d7dd6094308aeb7753d01b7ce304012e7f83d8d6ea5: Status 404 returned error can't find the container with id 68c2992804727179410d8d7dd6094308aeb7753d01b7ce304012e7f83d8d6ea5 Oct 30 22:12:17.073206 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:17.073169 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" event={"ID":"93fccba4-3698-413d-bd1a-11836e94bb20","Type":"ContainerStarted","Data":"78dfdbdf6ea43ca3581c7b61ec4390adcac05972429612a3b8cd81036e30fbeb"} Oct 30 22:12:17.073640 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:17.073225 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" event={"ID":"93fccba4-3698-413d-bd1a-11836e94bb20","Type":"ContainerStarted","Data":"68c2992804727179410d8d7dd6094308aeb7753d01b7ce304012e7f83d8d6ea5"} Oct 30 22:12:17.188369 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:17.188312 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" podStartSLOduration=2.188298158 podStartE2EDuration="2.188298158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:17.185807718 +0000 UTC m=+3327.319180920" watchObservedRunningTime="2026-10-30 22:12:17.188298158 +0000 UTC m=+3327.321671355" Oct 30 22:12:18.078902 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:18.078872 4526 generic.go:356] "Generic (PLEG): container finished" podID="93fccba4-3698-413d-bd1a-11836e94bb20" containerID="78dfdbdf6ea43ca3581c7b61ec4390adcac05972429612a3b8cd81036e30fbeb" exitCode=0 Oct 30 22:12:18.079317 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:18.078927 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" event={"ID":"93fccba4-3698-413d-bd1a-11836e94bb20","Type":"ContainerDied","Data":"78dfdbdf6ea43ca3581c7b61ec4390adcac05972429612a3b8cd81036e30fbeb"} Oct 30 22:12:19.216088 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.216061 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:19.381951 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.381861 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fccba4-3698-413d-bd1a-11836e94bb20-kube-api-access-fzrlc\" (UniqueName: \"kubernetes.io/projected/93fccba4-3698-413d-bd1a-11836e94bb20-kube-api-access-fzrlc\") pod \"93fccba4-3698-413d-bd1a-11836e94bb20\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " Oct 30 22:12:19.381951 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.381901 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-xks-ca\") pod \"93fccba4-3698-413d-bd1a-11836e94bb20\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " Oct 30 22:12:19.382184 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.381962 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-openshift-ca\") pod \"93fccba4-3698-413d-bd1a-11836e94bb20\" (UID: \"93fccba4-3698-413d-bd1a-11836e94bb20\") " Oct 30 22:12:19.382373 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.382340 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-openshift-ca" pod "93fccba4-3698-413d-bd1a-11836e94bb20" (UID: "93fccba4-3698-413d-bd1a-11836e94bb20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:19.384226 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.384204 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-xks-ca" pod "93fccba4-3698-413d-bd1a-11836e94bb20" (UID: "93fccba4-3698-413d-bd1a-11836e94bb20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:19.384375 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.384298 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fccba4-3698-413d-bd1a-11836e94bb20-kube-api-access-fzrlc" pod "93fccba4-3698-413d-bd1a-11836e94bb20" (UID: "93fccba4-3698-413d-bd1a-11836e94bb20"). InnerVolumeSpecName "kube-api-access-fzrlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:19.482990 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.482956 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzrlc\" (UniqueName: \"kubernetes.io/projected/93fccba4-3698-413d-bd1a-11836e94bb20-kube-api-access-fzrlc\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:19.482990 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.482986 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:19.482990 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:19.482997 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fccba4-3698-413d-bd1a-11836e94bb20-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:20.094240 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:20.094207 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" event={"ID":"93fccba4-3698-413d-bd1a-11836e94bb20","Type":"ContainerDied","Data":"68c2992804727179410d8d7dd6094308aeb7753d01b7ce304012e7f83d8d6ea5"} Oct 30 22:12:20.094421 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:20.094276 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c2992804727179410d8d7dd6094308aeb7753d01b7ce304012e7f83d8d6ea5" Oct 30 22:12:20.094421 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:20.094224 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q" Oct 30 22:12:26.450602 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.450568 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm"] Oct 30 22:12:26.451160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.450938 4526 cpu_manager.go:414] "containerMap: removing container" podUID="93fccba4-3698-413d-bd1a-11836e94bb20" containerName="revoke-keys" Oct 30 22:12:26.451160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.450948 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fccba4-3698-413d-bd1a-11836e94bb20" containerName="revoke-keys" Oct 30 22:12:26.451160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.451040 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fccba4-3698-413d-bd1a-11836e94bb20" containerName="revoke-keys" Oct 30 22:12:26.454252 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.454230 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.506325 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.506289 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:26.553847 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.553809 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.554047 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.553927 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.554047 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.554010 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd25w\" (UniqueName: \"kubernetes.io/projected/740a7c8a-e79b-4a07-82bb-69da6f042164-kube-api-access-xd25w\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.555628 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.555605 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:26.555747 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.555682 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:26.555747 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.555713 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:12:26.557915 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.557895 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm"] Oct 30 22:12:26.654615 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.654580 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.654853 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.654704 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd25w\" (UniqueName: \"kubernetes.io/projected/740a7c8a-e79b-4a07-82bb-69da6f042164-kube-api-access-xd25w\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.654853 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.654740 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.655253 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.655228 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.657224 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.657205 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.711841 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.711753 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd25w\" (UniqueName: \"kubernetes.io/projected/740a7c8a-e79b-4a07-82bb-69da6f042164-kube-api-access-xd25w\") pod \"maas-api-revoke-sub-5a29efb3-wh4lm\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:26.767382 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:26.767349 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:27.144006 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:27.143984 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm"] Oct 30 22:12:27.145956 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:12:27.145929 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740a7c8a_e79b_4a07_82bb_69da6f042164.slice/crio-52cc44f8c1c8f495af1992b47dcafcce908ceadeb3dbba9bf5cbaaa438b6dbc0 WatchSource:0}: Error finding container 52cc44f8c1c8f495af1992b47dcafcce908ceadeb3dbba9bf5cbaaa438b6dbc0: Status 404 returned error can't find the container with id 52cc44f8c1c8f495af1992b47dcafcce908ceadeb3dbba9bf5cbaaa438b6dbc0 Oct 30 22:12:28.132992 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:28.132954 4526 generic.go:356] "Generic (PLEG): container finished" podID="740a7c8a-e79b-4a07-82bb-69da6f042164" containerID="ed76c9831609c508fdb7c0f70b5877d8976492f2088e9e11b7ecf41de22d393a" exitCode=0 Oct 30 22:12:28.133409 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:28.133040 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" event={"ID":"740a7c8a-e79b-4a07-82bb-69da6f042164","Type":"ContainerDied","Data":"ed76c9831609c508fdb7c0f70b5877d8976492f2088e9e11b7ecf41de22d393a"} Oct 30 22:12:28.133409 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:28.133097 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" event={"ID":"740a7c8a-e79b-4a07-82bb-69da6f042164","Type":"ContainerStarted","Data":"52cc44f8c1c8f495af1992b47dcafcce908ceadeb3dbba9bf5cbaaa438b6dbc0"} Oct 30 22:12:29.300609 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.300577 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:29.383478 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.383441 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-xks-ca\") pod \"740a7c8a-e79b-4a07-82bb-69da6f042164\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " Oct 30 22:12:29.383720 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.383519 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740a7c8a-e79b-4a07-82bb-69da6f042164-kube-api-access-xd25w\" (UniqueName: \"kubernetes.io/projected/740a7c8a-e79b-4a07-82bb-69da6f042164-kube-api-access-xd25w\") pod \"740a7c8a-e79b-4a07-82bb-69da6f042164\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " Oct 30 22:12:29.383720 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.383611 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-openshift-ca\") pod \"740a7c8a-e79b-4a07-82bb-69da6f042164\" (UID: \"740a7c8a-e79b-4a07-82bb-69da6f042164\") " Oct 30 22:12:29.384279 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.384250 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-openshift-ca" pod "740a7c8a-e79b-4a07-82bb-69da6f042164" (UID: "740a7c8a-e79b-4a07-82bb-69da6f042164"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:29.385858 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.385829 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-xks-ca" pod "740a7c8a-e79b-4a07-82bb-69da6f042164" (UID: "740a7c8a-e79b-4a07-82bb-69da6f042164"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:29.386423 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.386400 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740a7c8a-e79b-4a07-82bb-69da6f042164-kube-api-access-xd25w" pod "740a7c8a-e79b-4a07-82bb-69da6f042164" (UID: "740a7c8a-e79b-4a07-82bb-69da6f042164"). InnerVolumeSpecName "kube-api-access-xd25w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:29.484714 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.484678 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:29.484714 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.484718 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd25w\" (UniqueName: \"kubernetes.io/projected/740a7c8a-e79b-4a07-82bb-69da6f042164-kube-api-access-xd25w\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:29.484937 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:29.484735 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/740a7c8a-e79b-4a07-82bb-69da6f042164-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:30.163569 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:30.162597 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" event={"ID":"740a7c8a-e79b-4a07-82bb-69da6f042164","Type":"ContainerDied","Data":"52cc44f8c1c8f495af1992b47dcafcce908ceadeb3dbba9bf5cbaaa438b6dbc0"} Oct 30 22:12:30.163569 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:30.162651 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52cc44f8c1c8f495af1992b47dcafcce908ceadeb3dbba9bf5cbaaa438b6dbc0" Oct 30 22:12:30.163569 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:30.162704 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm" Oct 30 22:12:42.045777 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:42.045736 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn"] Oct 30 22:12:42.095150 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:42.095115 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-6frxn"] Oct 30 22:12:42.476510 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:12:42.476395 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:12:43.984115 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:43.984076 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a63b28-5721-4031-8122-b8ef2c0c2f46" path="/var/lib/kubelet/pods/09a63b28-5721-4031-8122-b8ef2c0c2f46/volumes" Oct 30 22:12:48.862567 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:48.862526 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc"] Oct 30 22:12:48.911967 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:48.911934 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5bvqc"] Oct 30 22:12:48.985237 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:48.985198 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn"] Oct 30 22:12:48.985594 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:48.985581 4526 cpu_manager.go:414] "containerMap: removing container" podUID="740a7c8a-e79b-4a07-82bb-69da6f042164" containerName="revoke-keys" Oct 30 22:12:48.985649 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:48.985595 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740a7c8a-e79b-4a07-82bb-69da6f042164" containerName="revoke-keys" Oct 30 22:12:48.985684 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:48.985667 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="740a7c8a-e79b-4a07-82bb-69da6f042164" containerName="revoke-keys" Oct 30 22:12:48.989020 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:48.989005 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.042201 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.042173 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzck4\" (UniqueName: \"kubernetes.io/projected/676162fc-188b-4201-96db-33bf3efd86dd-kube-api-access-lzck4\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.042349 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.042213 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.042349 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.042319 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.044446 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.044366 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:49.044606 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.044565 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:49.044842 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.044798 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:49.044986 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.044896 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:12:49.047923 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.047902 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn"] Oct 30 22:12:49.142844 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.142750 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzck4\" (UniqueName: \"kubernetes.io/projected/676162fc-188b-4201-96db-33bf3efd86dd-kube-api-access-lzck4\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.142844 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.142797 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.143077 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.142868 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.143559 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.143540 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.145336 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.145316 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.199226 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.199187 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzck4\" (UniqueName: \"kubernetes.io/projected/676162fc-188b-4201-96db-33bf3efd86dd-kube-api-access-lzck4\") pod \"maas-api-revoke-sub-3d4156f1-7zfvn\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.301551 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.301514 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:49.676824 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.676794 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn"] Oct 30 22:12:49.678961 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:12:49.678930 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676162fc_188b_4201_96db_33bf3efd86dd.slice/crio-7a8369121245220f30b5b84d04ba5747f087b76d50a048ad6251a9e5cf8a0bb2 WatchSource:0}: Error finding container 7a8369121245220f30b5b84d04ba5747f087b76d50a048ad6251a9e5cf8a0bb2: Status 404 returned error can't find the container with id 7a8369121245220f30b5b84d04ba5747f087b76d50a048ad6251a9e5cf8a0bb2 Oct 30 22:12:49.983934 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:49.983858 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a5a772-193d-4599-afe5-0aee07c59935" path="/var/lib/kubelet/pods/69a5a772-193d-4599-afe5-0aee07c59935/volumes" Oct 30 22:12:50.276079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:50.275982 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" event={"ID":"676162fc-188b-4201-96db-33bf3efd86dd","Type":"ContainerStarted","Data":"88abe9154087eab1d7ca37f3f652f74e8be23475990b888dd008fc35710f743b"} Oct 30 22:12:50.276079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:50.276048 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" event={"ID":"676162fc-188b-4201-96db-33bf3efd86dd","Type":"ContainerStarted","Data":"7a8369121245220f30b5b84d04ba5747f087b76d50a048ad6251a9e5cf8a0bb2"} Oct 30 22:12:50.389319 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:50.389262 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" podStartSLOduration=2.389245637 podStartE2EDuration="2.389245637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:50.387762448 +0000 UTC m=+3360.521135647" watchObservedRunningTime="2026-10-30 22:12:50.389245637 +0000 UTC m=+3360.522618838" Oct 30 22:12:51.282429 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:51.282394 4526 generic.go:356] "Generic (PLEG): container finished" podID="676162fc-188b-4201-96db-33bf3efd86dd" containerID="88abe9154087eab1d7ca37f3f652f74e8be23475990b888dd008fc35710f743b" exitCode=0 Oct 30 22:12:51.282848 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:51.282480 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" event={"ID":"676162fc-188b-4201-96db-33bf3efd86dd","Type":"ContainerDied","Data":"88abe9154087eab1d7ca37f3f652f74e8be23475990b888dd008fc35710f743b"} Oct 30 22:12:52.421163 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.421139 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:52.472612 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.472572 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676162fc-188b-4201-96db-33bf3efd86dd-kube-api-access-lzck4\" (UniqueName: \"kubernetes.io/projected/676162fc-188b-4201-96db-33bf3efd86dd-kube-api-access-lzck4\") pod \"676162fc-188b-4201-96db-33bf3efd86dd\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " Oct 30 22:12:52.472803 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.472619 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-xks-ca\") pod \"676162fc-188b-4201-96db-33bf3efd86dd\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " Oct 30 22:12:52.472803 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.472706 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-openshift-ca\") pod \"676162fc-188b-4201-96db-33bf3efd86dd\" (UID: \"676162fc-188b-4201-96db-33bf3efd86dd\") " Oct 30 22:12:52.473117 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.473080 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-openshift-ca" pod "676162fc-188b-4201-96db-33bf3efd86dd" (UID: "676162fc-188b-4201-96db-33bf3efd86dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:52.475020 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.474995 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-xks-ca" pod "676162fc-188b-4201-96db-33bf3efd86dd" (UID: "676162fc-188b-4201-96db-33bf3efd86dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:52.475121 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.475077 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676162fc-188b-4201-96db-33bf3efd86dd-kube-api-access-lzck4" pod "676162fc-188b-4201-96db-33bf3efd86dd" (UID: "676162fc-188b-4201-96db-33bf3efd86dd"). InnerVolumeSpecName "kube-api-access-lzck4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:52.574137 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.574055 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzck4\" (UniqueName: \"kubernetes.io/projected/676162fc-188b-4201-96db-33bf3efd86dd-kube-api-access-lzck4\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.574137 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.574081 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:52.574137 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:52.574092 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676162fc-188b-4201-96db-33bf3efd86dd-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:53.292898 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:53.292867 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" event={"ID":"676162fc-188b-4201-96db-33bf3efd86dd","Type":"ContainerDied","Data":"7a8369121245220f30b5b84d04ba5747f087b76d50a048ad6251a9e5cf8a0bb2"} Oct 30 22:12:53.293064 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:53.292904 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn" Oct 30 22:12:53.293064 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:53.292915 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a8369121245220f30b5b84d04ba5747f087b76d50a048ad6251a9e5cf8a0bb2" Oct 30 22:12:54.047343 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:54.047306 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh"] Oct 30 22:12:54.096718 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:54.096684 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dpzxh"] Oct 30 22:12:55.983394 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:12:55.983359 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcf6f6a-87ef-4ec7-a3f9-8130d1295a46" path="/var/lib/kubelet/pods/afcf6f6a-87ef-4ec7-a3f9-8130d1295a46/volumes" Oct 30 22:13:12.477306 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:13:12.477267 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:13:29.046590 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:29.046558 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j"] Oct 30 22:13:29.095636 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:29.095606 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pxz6j"] Oct 30 22:13:29.983365 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:29.983329 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5196f6-c994-41ee-a330-06766bb590ab" path="/var/lib/kubelet/pods/8a5196f6-c994-41ee-a330-06766bb590ab/volumes" Oct 30 22:13:40.052454 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:40.052419 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc"] Oct 30 22:13:40.101437 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:40.101402 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-hx8cc"] Oct 30 22:13:41.983744 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:41.983714 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6" path="/var/lib/kubelet/pods/fd6ac7b7-989e-4bfc-bfb8-b3b467232fe6/volumes" Oct 30 22:13:42.478062 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:13:42.478027 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:13:53.035781 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:53.035749 4526 scope.go:122] "RemoveContainer" containerID="dcbbe767e3f1977bea89226851b46fe9578e66b1a7ab0045daa761b3db4ae280" Oct 30 22:13:53.045045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:53.045019 4526 scope.go:122] "RemoveContainer" containerID="1b7247626bc8d965390bdefcc94efb6ca93551f496f1604a69c5d05a25821c06" Oct 30 22:13:53.055442 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:13:53.055420 4526 scope.go:122] "RemoveContainer" containerID="2eeedbf05ca9fa07ad04f96e91b061191b32f9780a1ceaafeb9bec1a3162936a" Oct 30 22:14:12.477643 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:14:12.477603 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:14:15.045709 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:15.045670 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422"] Oct 30 22:14:15.094783 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:15.094750 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nx422"] Oct 30 22:14:15.983141 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:15.983104 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45bb786-2602-4b18-8e7a-6c19d9978ba0" path="/var/lib/kubelet/pods/c45bb786-2602-4b18-8e7a-6c19d9978ba0/volumes" Oct 30 22:14:42.482015 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:14:42.481932 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:14:43.046462 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:43.046431 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t"] Oct 30 22:14:43.095373 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:43.095340 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-94f9t"] Oct 30 22:14:43.983680 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:43.983647 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856efb22-d65c-4f66-a33e-7ffd3605deed" path="/var/lib/kubelet/pods/856efb22-d65c-4f66-a33e-7ffd3605deed/volumes" Oct 30 22:14:53.081538 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:53.081507 4526 scope.go:122] "RemoveContainer" containerID="51039831c24e835612183e90998b03f82c045f97e30b6d7b8d974ee2c5af0d2a" Oct 30 22:14:53.089680 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:53.089657 4526 scope.go:122] "RemoveContainer" containerID="75f482522186cefad74c5f898ca68e998669485d2006d8c0f5dfd9fdf3ddee5a" Oct 30 22:14:54.175311 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.175282 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs"] Oct 30 22:14:54.175736 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.175669 4526 cpu_manager.go:414] "containerMap: removing container" podUID="676162fc-188b-4201-96db-33bf3efd86dd" containerName="revoke-keys" Oct 30 22:14:54.175736 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.175682 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676162fc-188b-4201-96db-33bf3efd86dd" containerName="revoke-keys" Oct 30 22:14:54.175816 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.175758 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="676162fc-188b-4201-96db-33bf3efd86dd" containerName="revoke-keys" Oct 30 22:14:54.178714 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.178697 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.231755 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.231728 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:54.280661 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.280620 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:54.280661 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.280637 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:14:54.280940 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.280814 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:54.282872 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.282845 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs"] Oct 30 22:14:54.313967 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.313941 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7m8\" (UniqueName: \"kubernetes.io/projected/a5da7764-04f5-45ad-910b-f367ec758e2d-kube-api-access-tl7m8\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.314111 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.313978 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.314111 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.313999 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.414583 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.414543 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7m8\" (UniqueName: \"kubernetes.io/projected/a5da7764-04f5-45ad-910b-f367ec758e2d-kube-api-access-tl7m8\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.414583 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.414587 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.414811 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.414604 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.415233 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.415214 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.417159 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.417139 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.472122 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.472045 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7m8\" (UniqueName: \"kubernetes.io/projected/a5da7764-04f5-45ad-910b-f367ec758e2d-kube-api-access-tl7m8\") pod \"maas-api-revoke-sub-d802c957-cbncs\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.490872 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.490842 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:54.862483 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.862460 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs"] Oct 30 22:14:54.865003 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:14:54.864970 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5da7764_04f5_45ad_910b_f367ec758e2d.slice/crio-106b6a5b41730808820575d2b5ae3d2345e33c30edb491c8f1f946f7e59fb70d WatchSource:0}: Error finding container 106b6a5b41730808820575d2b5ae3d2345e33c30edb491c8f1f946f7e59fb70d: Status 404 returned error can't find the container with id 106b6a5b41730808820575d2b5ae3d2345e33c30edb491c8f1f946f7e59fb70d Oct 30 22:14:54.881641 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:54.881601 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" event={"ID":"a5da7764-04f5-45ad-910b-f367ec758e2d","Type":"ContainerStarted","Data":"106b6a5b41730808820575d2b5ae3d2345e33c30edb491c8f1f946f7e59fb70d"} Oct 30 22:14:55.888230 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:55.888188 4526 generic.go:356] "Generic (PLEG): container finished" podID="a5da7764-04f5-45ad-910b-f367ec758e2d" containerID="161b2b980a83331553f651ec919057d73a201cc34d4d9afa7bbcfb452973a750" exitCode=0 Oct 30 22:14:55.888660 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:55.888235 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" event={"ID":"a5da7764-04f5-45ad-910b-f367ec758e2d","Type":"ContainerDied","Data":"161b2b980a83331553f651ec919057d73a201cc34d4d9afa7bbcfb452973a750"} Oct 30 22:14:57.021340 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.021317 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:14:57.140185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.140158 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5da7764-04f5-45ad-910b-f367ec758e2d-kube-api-access-tl7m8\" (UniqueName: \"kubernetes.io/projected/a5da7764-04f5-45ad-910b-f367ec758e2d-kube-api-access-tl7m8\") pod \"a5da7764-04f5-45ad-910b-f367ec758e2d\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " Oct 30 22:14:57.140363 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.140256 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-xks-ca\") pod \"a5da7764-04f5-45ad-910b-f367ec758e2d\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " Oct 30 22:14:57.140363 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.140309 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-openshift-ca\") pod \"a5da7764-04f5-45ad-910b-f367ec758e2d\" (UID: \"a5da7764-04f5-45ad-910b-f367ec758e2d\") " Oct 30 22:14:57.140780 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.140749 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-openshift-ca" pod "a5da7764-04f5-45ad-910b-f367ec758e2d" (UID: "a5da7764-04f5-45ad-910b-f367ec758e2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:57.142475 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.142449 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-xks-ca" pod "a5da7764-04f5-45ad-910b-f367ec758e2d" (UID: "a5da7764-04f5-45ad-910b-f367ec758e2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:57.142718 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.142694 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5da7764-04f5-45ad-910b-f367ec758e2d-kube-api-access-tl7m8" pod "a5da7764-04f5-45ad-910b-f367ec758e2d" (UID: "a5da7764-04f5-45ad-910b-f367ec758e2d"). InnerVolumeSpecName "kube-api-access-tl7m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:57.241825 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.241792 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl7m8\" (UniqueName: \"kubernetes.io/projected/a5da7764-04f5-45ad-910b-f367ec758e2d-kube-api-access-tl7m8\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:57.241825 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.241818 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:57.241825 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.241828 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5da7764-04f5-45ad-910b-f367ec758e2d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:57.899840 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.899801 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" event={"ID":"a5da7764-04f5-45ad-910b-f367ec758e2d","Type":"ContainerDied","Data":"106b6a5b41730808820575d2b5ae3d2345e33c30edb491c8f1f946f7e59fb70d"} Oct 30 22:14:57.900000 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.899855 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106b6a5b41730808820575d2b5ae3d2345e33c30edb491c8f1f946f7e59fb70d" Oct 30 22:14:57.900000 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:14:57.899817 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs" Oct 30 22:15:00.159525 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.159475 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng"] Oct 30 22:15:00.160021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.159855 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a5da7764-04f5-45ad-910b-f367ec758e2d" containerName="revoke-keys" Oct 30 22:15:00.160021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.159867 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5da7764-04f5-45ad-910b-f367ec758e2d" containerName="revoke-keys" Oct 30 22:15:00.160021 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.159950 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5da7764-04f5-45ad-910b-f367ec758e2d" containerName="revoke-keys" Oct 30 22:15:00.163039 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.163022 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:00.217045 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.217016 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:00.266171 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.266140 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:00.266171 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.266138 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:15:00.268249 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.268222 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng"] Oct 30 22:15:00.270092 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.270071 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de8347c-98a0-47c0-aedb-496a981cddb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng\" (UID: \"7de8347c-98a0-47c0-aedb-496a981cddb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:00.270211 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.270117 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtj2m\" (UniqueName: \"kubernetes.io/projected/7de8347c-98a0-47c0-aedb-496a981cddb3-kube-api-access-rtj2m\") pod \"maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng\" (UID: \"7de8347c-98a0-47c0-aedb-496a981cddb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:00.371773 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.371741 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de8347c-98a0-47c0-aedb-496a981cddb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng\" (UID: \"7de8347c-98a0-47c0-aedb-496a981cddb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:00.371959 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.371786 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtj2m\" (UniqueName: \"kubernetes.io/projected/7de8347c-98a0-47c0-aedb-496a981cddb3-kube-api-access-rtj2m\") pod \"maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng\" (UID: \"7de8347c-98a0-47c0-aedb-496a981cddb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:00.372355 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.372335 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de8347c-98a0-47c0-aedb-496a981cddb3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng\" (UID: \"7de8347c-98a0-47c0-aedb-496a981cddb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:00.429977 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.429908 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtj2m\" (UniqueName: \"kubernetes.io/projected/7de8347c-98a0-47c0-aedb-496a981cddb3-kube-api-access-rtj2m\") pod \"maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng\" (UID: \"7de8347c-98a0-47c0-aedb-496a981cddb3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:00.475680 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.475650 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:00.846821 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.846789 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng"] Oct 30 22:15:00.850087 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:15:00.850061 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de8347c_98a0_47c0_aedb_496a981cddb3.slice/crio-5bdff963d58af9c30a2500a4aaa8e71f8aebdeb4f3e3fcafa29ad1c473fb5d1a WatchSource:0}: Error finding container 5bdff963d58af9c30a2500a4aaa8e71f8aebdeb4f3e3fcafa29ad1c473fb5d1a: Status 404 returned error can't find the container with id 5bdff963d58af9c30a2500a4aaa8e71f8aebdeb4f3e3fcafa29ad1c473fb5d1a Oct 30 22:15:00.914617 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:00.914586 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" event={"ID":"7de8347c-98a0-47c0-aedb-496a981cddb3","Type":"ContainerStarted","Data":"5bdff963d58af9c30a2500a4aaa8e71f8aebdeb4f3e3fcafa29ad1c473fb5d1a"} Oct 30 22:15:01.921587 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:01.921464 4526 generic.go:356] "Generic (PLEG): container finished" podID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerID="54f242c566b01052c5c5248183c14950626df085b93c1723dcf8a3e2561a8ffb" exitCode=60 Oct 30 22:15:01.921587 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:01.921558 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" event={"ID":"7de8347c-98a0-47c0-aedb-496a981cddb3","Type":"ContainerDied","Data":"54f242c566b01052c5c5248183c14950626df085b93c1723dcf8a3e2561a8ffb"} Oct 30 22:15:01.922005 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:01.921796 4526 scope.go:122] "RemoveContainer" containerID="54f242c566b01052c5c5248183c14950626df085b93c1723dcf8a3e2561a8ffb" Oct 30 22:15:02.928257 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:02.928223 4526 generic.go:356] "Generic (PLEG): container finished" podID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerID="1a3a813154186190d6f0486f97c6aed58f223cb7f180892a9d68852e82aeb8cc" exitCode=60 Oct 30 22:15:02.928721 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:02.928306 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" event={"ID":"7de8347c-98a0-47c0-aedb-496a981cddb3","Type":"ContainerDied","Data":"1a3a813154186190d6f0486f97c6aed58f223cb7f180892a9d68852e82aeb8cc"} Oct 30 22:15:02.928721 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:02.928394 4526 scope.go:122] "RemoveContainer" containerID="54f242c566b01052c5c5248183c14950626df085b93c1723dcf8a3e2561a8ffb" Oct 30 22:15:02.928721 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:02.928641 4526 scope.go:122] "RemoveContainer" containerID="1a3a813154186190d6f0486f97c6aed58f223cb7f180892a9d68852e82aeb8cc" Oct 30 22:15:02.928905 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:15:02.928882 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng_odh-ai-gateway-infra(7de8347c-98a0-47c0-aedb-496a981cddb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" Oct 30 22:15:03.935310 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:03.935284 4526 scope.go:122] "RemoveContainer" containerID="1a3a813154186190d6f0486f97c6aed58f223cb7f180892a9d68852e82aeb8cc" Oct 30 22:15:03.935768 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:15:03.935558 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng_odh-ai-gateway-infra(7de8347c-98a0-47c0-aedb-496a981cddb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" Oct 30 22:15:06.553305 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.553267 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj"] Oct 30 22:15:06.558102 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.558077 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.615692 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.615655 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj"] Oct 30 22:15:06.631061 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.631017 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.631061 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.631064 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.631306 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.631166 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgd82\" (UniqueName: \"kubernetes.io/projected/c034e5e9-196f-48cf-a277-7a35c599a91f-kube-api-access-jgd82\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.658856 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.658826 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:06.732329 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.732288 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgd82\" (UniqueName: \"kubernetes.io/projected/c034e5e9-196f-48cf-a277-7a35c599a91f-kube-api-access-jgd82\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.732566 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.732470 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.732675 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.732603 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.733086 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.733066 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.735058 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.735035 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.789779 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.789745 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgd82\" (UniqueName: \"kubernetes.io/projected/c034e5e9-196f-48cf-a277-7a35c599a91f-kube-api-access-jgd82\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:06.871732 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:06.871638 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:07.245680 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:07.245651 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj"] Oct 30 22:15:07.248105 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:15:07.248079 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc034e5e9_196f_48cf_a277_7a35c599a91f.slice/crio-f6e828ee22bcc8d227173aea416bd155492286b31ae919d9fdaa1fcf42268fa3 WatchSource:0}: Error finding container f6e828ee22bcc8d227173aea416bd155492286b31ae919d9fdaa1fcf42268fa3: Status 404 returned error can't find the container with id f6e828ee22bcc8d227173aea416bd155492286b31ae919d9fdaa1fcf42268fa3 Oct 30 22:15:07.955718 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:07.955681 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" event={"ID":"c034e5e9-196f-48cf-a277-7a35c599a91f","Type":"ContainerStarted","Data":"1d3441fd2f65e0e4387e5c4dd1f4e502391c6fa58307200104fb773004711299"} Oct 30 22:15:07.956150 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:07.955734 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" event={"ID":"c034e5e9-196f-48cf-a277-7a35c599a91f","Type":"ContainerStarted","Data":"f6e828ee22bcc8d227173aea416bd155492286b31ae919d9fdaa1fcf42268fa3"} Oct 30 22:15:08.069394 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:08.069340 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" podStartSLOduration=2.069324782 podStartE2EDuration="2.069324782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:08.067907435 +0000 UTC m=+3498.201280632" watchObservedRunningTime="2026-10-30 22:15:08.069324782 +0000 UTC m=+3498.202698005" Oct 30 22:15:08.962639 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:08.962601 4526 generic.go:356] "Generic (PLEG): container finished" podID="c034e5e9-196f-48cf-a277-7a35c599a91f" containerID="1d3441fd2f65e0e4387e5c4dd1f4e502391c6fa58307200104fb773004711299" exitCode=0 Oct 30 22:15:08.963118 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:08.962688 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" event={"ID":"c034e5e9-196f-48cf-a277-7a35c599a91f","Type":"ContainerDied","Data":"1d3441fd2f65e0e4387e5c4dd1f4e502391c6fa58307200104fb773004711299"} Oct 30 22:15:10.102682 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.102659 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:10.157452 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.157421 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-xks-ca\") pod \"c034e5e9-196f-48cf-a277-7a35c599a91f\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " Oct 30 22:15:10.157669 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.157479 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c034e5e9-196f-48cf-a277-7a35c599a91f-kube-api-access-jgd82\" (UniqueName: \"kubernetes.io/projected/c034e5e9-196f-48cf-a277-7a35c599a91f-kube-api-access-jgd82\") pod \"c034e5e9-196f-48cf-a277-7a35c599a91f\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " Oct 30 22:15:10.159724 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.159702 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-xks-ca" pod "c034e5e9-196f-48cf-a277-7a35c599a91f" (UID: "c034e5e9-196f-48cf-a277-7a35c599a91f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:10.159850 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.159728 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c034e5e9-196f-48cf-a277-7a35c599a91f-kube-api-access-jgd82" pod "c034e5e9-196f-48cf-a277-7a35c599a91f" (UID: "c034e5e9-196f-48cf-a277-7a35c599a91f"). InnerVolumeSpecName "kube-api-access-jgd82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:10.258920 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.258850 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-openshift-ca\") pod \"c034e5e9-196f-48cf-a277-7a35c599a91f\" (UID: \"c034e5e9-196f-48cf-a277-7a35c599a91f\") " Oct 30 22:15:10.259052 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.259042 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.259100 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.259053 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgd82\" (UniqueName: \"kubernetes.io/projected/c034e5e9-196f-48cf-a277-7a35c599a91f-kube-api-access-jgd82\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.259176 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.259154 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-openshift-ca" pod "c034e5e9-196f-48cf-a277-7a35c599a91f" (UID: "c034e5e9-196f-48cf-a277-7a35c599a91f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:10.359967 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.359936 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c034e5e9-196f-48cf-a277-7a35c599a91f-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:10.974786 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.974747 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" Oct 30 22:15:10.974960 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.974751 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj" event={"ID":"c034e5e9-196f-48cf-a277-7a35c599a91f","Type":"ContainerDied","Data":"f6e828ee22bcc8d227173aea416bd155492286b31ae919d9fdaa1fcf42268fa3"} Oct 30 22:15:10.974960 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:10.974877 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6e828ee22bcc8d227173aea416bd155492286b31ae919d9fdaa1fcf42268fa3" Oct 30 22:15:12.477035 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:15:12.476998 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:15:16.978841 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:16.978809 4526 scope.go:122] "RemoveContainer" containerID="1a3a813154186190d6f0486f97c6aed58f223cb7f180892a9d68852e82aeb8cc" Oct 30 22:15:18.014458 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:18.014429 4526 generic.go:356] "Generic (PLEG): container finished" podID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerID="8ba73a34eb4c3ba6f94f5e4458e792f3a288c6b9c55f98a9d5cd1638175110a5" exitCode=60 Oct 30 22:15:18.014913 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:18.014521 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" event={"ID":"7de8347c-98a0-47c0-aedb-496a981cddb3","Type":"ContainerDied","Data":"8ba73a34eb4c3ba6f94f5e4458e792f3a288c6b9c55f98a9d5cd1638175110a5"} Oct 30 22:15:18.014913 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:18.014583 4526 scope.go:122] "RemoveContainer" containerID="1a3a813154186190d6f0486f97c6aed58f223cb7f180892a9d68852e82aeb8cc" Oct 30 22:15:18.015036 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:18.014916 4526 scope.go:122] "RemoveContainer" containerID="8ba73a34eb4c3ba6f94f5e4458e792f3a288c6b9c55f98a9d5cd1638175110a5" Oct 30 22:15:18.015206 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:15:18.015186 4526 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng_odh-ai-gateway-infra(7de8347c-98a0-47c0-aedb-496a981cddb3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" Oct 30 22:15:19.146483 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:19.146449 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng"] Oct 30 22:15:19.291330 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:19.291305 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:19.338382 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:19.338343 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de8347c-98a0-47c0-aedb-496a981cddb3-kube-api-access-rtj2m\" (UniqueName: \"kubernetes.io/projected/7de8347c-98a0-47c0-aedb-496a981cddb3-kube-api-access-rtj2m\") pod \"7de8347c-98a0-47c0-aedb-496a981cddb3\" (UID: \"7de8347c-98a0-47c0-aedb-496a981cddb3\") " Oct 30 22:15:19.338595 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:19.338415 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de8347c-98a0-47c0-aedb-496a981cddb3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de8347c-98a0-47c0-aedb-496a981cddb3-maas-api-service-ca\") pod \"7de8347c-98a0-47c0-aedb-496a981cddb3\" (UID: \"7de8347c-98a0-47c0-aedb-496a981cddb3\") " Oct 30 22:15:19.338793 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:19.338771 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de8347c-98a0-47c0-aedb-496a981cddb3-maas-api-service-ca" pod "7de8347c-98a0-47c0-aedb-496a981cddb3" (UID: "7de8347c-98a0-47c0-aedb-496a981cddb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.340692 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:19.340665 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de8347c-98a0-47c0-aedb-496a981cddb3-kube-api-access-rtj2m" pod "7de8347c-98a0-47c0-aedb-496a981cddb3" (UID: "7de8347c-98a0-47c0-aedb-496a981cddb3"). InnerVolumeSpecName "kube-api-access-rtj2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.439424 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:19.439331 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de8347c-98a0-47c0-aedb-496a981cddb3-maas-api-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.439424 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:19.439370 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtj2m\" (UniqueName: \"kubernetes.io/projected/7de8347c-98a0-47c0-aedb-496a981cddb3-kube-api-access-rtj2m\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.027121 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:20.027089 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" Oct 30 22:15:20.027327 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:20.027085 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng" event={"ID":"7de8347c-98a0-47c0-aedb-496a981cddb3","Type":"ContainerDied","Data":"5bdff963d58af9c30a2500a4aaa8e71f8aebdeb4f3e3fcafa29ad1c473fb5d1a"} Oct 30 22:15:20.027327 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:20.027228 4526 scope.go:122] "RemoveContainer" containerID="8ba73a34eb4c3ba6f94f5e4458e792f3a288c6b9c55f98a9d5cd1638175110a5" Oct 30 22:15:20.191180 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:20.191151 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng"] Oct 30 22:15:20.246115 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:20.242746 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-c3e01b-2988997r4lng"] Oct 30 22:15:21.983501 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:21.983453 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" path="/var/lib/kubelet/pods/7de8347c-98a0-47c0-aedb-496a981cddb3/volumes" Oct 30 22:15:26.163275 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:26.163239 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj"] Oct 30 22:15:26.212353 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:26.212318 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3e01b-44wfj"] Oct 30 22:15:27.983524 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:27.983475 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c034e5e9-196f-48cf-a277-7a35c599a91f" path="/var/lib/kubelet/pods/c034e5e9-196f-48cf-a277-7a35c599a91f/volumes" Oct 30 22:15:28.095472 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:28.095440 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6"] Oct 30 22:15:28.132315 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:28.132283 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rxpb6"] Oct 30 22:15:29.985090 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:29.985055 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76fbb69-8ad2-4d44-adc5-bbde503edc46" path="/var/lib/kubelet/pods/a76fbb69-8ad2-4d44-adc5-bbde503edc46/volumes" Oct 30 22:15:34.544731 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.544702 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn"] Oct 30 22:15:34.594162 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.594128 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7zfvn"] Oct 30 22:15:34.700448 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700409 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws"] Oct 30 22:15:34.700899 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700881 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:15:34.701002 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700901 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:15:34.701002 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700917 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c034e5e9-196f-48cf-a277-7a35c599a91f" containerName="revoke-keys" Oct 30 22:15:34.701002 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700924 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c034e5e9-196f-48cf-a277-7a35c599a91f" containerName="revoke-keys" Oct 30 22:15:34.701002 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700944 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:15:34.701002 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700952 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:15:34.701002 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700976 4526 cpu_manager.go:414] "containerMap: removing container" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:15:34.701002 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.700982 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:15:34.701371 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.701074 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:15:34.701371 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.701091 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c034e5e9-196f-48cf-a277-7a35c599a91f" containerName="revoke-keys" Oct 30 22:15:34.701371 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.701102 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:15:34.705369 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.705348 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.757702 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.757673 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:34.758083 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.758068 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:34.760707 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.760691 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:15:34.791483 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.791439 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.791712 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.791569 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.791712 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.791683 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8zm\" (UniqueName: \"kubernetes.io/projected/9d52084f-c32f-4822-845e-15cdf2536113-kube-api-access-zc8zm\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.806820 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.806751 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:34.808731 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.808700 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws"] Oct 30 22:15:34.892840 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.892803 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8zm\" (UniqueName: \"kubernetes.io/projected/9d52084f-c32f-4822-845e-15cdf2536113-kube-api-access-zc8zm\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.893067 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.892889 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.893067 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.892937 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.893649 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.893625 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.895580 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.895560 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:34.950702 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:34.950669 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8zm\" (UniqueName: \"kubernetes.io/projected/9d52084f-c32f-4822-845e-15cdf2536113-kube-api-access-zc8zm\") pod \"maas-api-revoke-sub-3d4156f1-s7mws\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:35.018130 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:35.018090 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:35.392712 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:35.392684 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws"] Oct 30 22:15:35.395708 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:15:35.395680 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d52084f_c32f_4822_845e_15cdf2536113.slice/crio-f7a966e62a4e6cc7091de825da7f9d8cf17cfd30a28095993b820b2d13ebd89e WatchSource:0}: Error finding container f7a966e62a4e6cc7091de825da7f9d8cf17cfd30a28095993b820b2d13ebd89e: Status 404 returned error can't find the container with id f7a966e62a4e6cc7091de825da7f9d8cf17cfd30a28095993b820b2d13ebd89e Oct 30 22:15:35.983429 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:35.983352 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676162fc-188b-4201-96db-33bf3efd86dd" path="/var/lib/kubelet/pods/676162fc-188b-4201-96db-33bf3efd86dd/volumes" Oct 30 22:15:36.106812 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:36.106779 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" event={"ID":"9d52084f-c32f-4822-845e-15cdf2536113","Type":"ContainerStarted","Data":"678efbdf095f187b9bab256a9588d0ff508845a62ed83fa11b1d2ce996a6e5ce"} Oct 30 22:15:36.106987 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:36.106833 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" event={"ID":"9d52084f-c32f-4822-845e-15cdf2536113","Type":"ContainerStarted","Data":"f7a966e62a4e6cc7091de825da7f9d8cf17cfd30a28095993b820b2d13ebd89e"} Oct 30 22:15:36.224529 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:36.224458 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" podStartSLOduration=2.224444913 podStartE2EDuration="2.224444913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:36.221212481 +0000 UTC m=+3526.354585679" watchObservedRunningTime="2026-10-30 22:15:36.224444913 +0000 UTC m=+3526.357818105" Oct 30 22:15:37.113529 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:37.113477 4526 generic.go:356] "Generic (PLEG): container finished" podID="9d52084f-c32f-4822-845e-15cdf2536113" containerID="678efbdf095f187b9bab256a9588d0ff508845a62ed83fa11b1d2ce996a6e5ce" exitCode=0 Oct 30 22:15:37.113529 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:37.113520 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" event={"ID":"9d52084f-c32f-4822-845e-15cdf2536113","Type":"ContainerDied","Data":"678efbdf095f187b9bab256a9588d0ff508845a62ed83fa11b1d2ce996a6e5ce"} Oct 30 22:15:38.253922 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.253898 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:38.327182 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.327147 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d52084f-c32f-4822-845e-15cdf2536113-kube-api-access-zc8zm\" (UniqueName: \"kubernetes.io/projected/9d52084f-c32f-4822-845e-15cdf2536113-kube-api-access-zc8zm\") pod \"9d52084f-c32f-4822-845e-15cdf2536113\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " Oct 30 22:15:38.327380 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.327193 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-xks-ca\") pod \"9d52084f-c32f-4822-845e-15cdf2536113\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " Oct 30 22:15:38.327380 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.327273 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-openshift-ca\") pod \"9d52084f-c32f-4822-845e-15cdf2536113\" (UID: \"9d52084f-c32f-4822-845e-15cdf2536113\") " Oct 30 22:15:38.327747 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.327718 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-openshift-ca" pod "9d52084f-c32f-4822-845e-15cdf2536113" (UID: "9d52084f-c32f-4822-845e-15cdf2536113"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:38.329532 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.329505 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-xks-ca" pod "9d52084f-c32f-4822-845e-15cdf2536113" (UID: "9d52084f-c32f-4822-845e-15cdf2536113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:38.329635 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.329621 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d52084f-c32f-4822-845e-15cdf2536113-kube-api-access-zc8zm" pod "9d52084f-c32f-4822-845e-15cdf2536113" (UID: "9d52084f-c32f-4822-845e-15cdf2536113"). InnerVolumeSpecName "kube-api-access-zc8zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:38.428878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.428804 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.428878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.428833 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8zm\" (UniqueName: \"kubernetes.io/projected/9d52084f-c32f-4822-845e-15cdf2536113-kube-api-access-zc8zm\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.428878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:38.428843 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d52084f-c32f-4822-845e-15cdf2536113-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:39.125833 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:39.125808 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" Oct 30 22:15:39.126017 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:39.125844 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws" event={"ID":"9d52084f-c32f-4822-845e-15cdf2536113","Type":"ContainerDied","Data":"f7a966e62a4e6cc7091de825da7f9d8cf17cfd30a28095993b820b2d13ebd89e"} Oct 30 22:15:39.126017 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:39.125891 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7a966e62a4e6cc7091de825da7f9d8cf17cfd30a28095993b820b2d13ebd89e" Oct 30 22:15:42.477344 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:15:42.477305 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:15:52.047948 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:52.047910 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4"] Oct 30 22:15:52.097345 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:52.097307 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptkz4"] Oct 30 22:15:53.114550 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:53.114515 4526 scope.go:122] "RemoveContainer" containerID="d09614d084cb958b1e525c34e489fec8e04022c66b1385f5dc3440c357b66a69" Oct 30 22:15:53.129373 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:53.129350 4526 scope.go:122] "RemoveContainer" containerID="21cb6ae2cb3db24d3b31595bc37b3ea3f850a9ed05e8d94810ad1624b634641c" Oct 30 22:15:53.983804 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:15:53.983765 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ba8542-615d-4c84-9f99-7d0416e2f258" path="/var/lib/kubelet/pods/c1ba8542-615d-4c84-9f99-7d0416e2f258/volumes" Oct 30 22:16:12.478520 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:16:12.478458 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:16:42.477695 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:16:42.477658 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:16:53.157469 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:16:53.157437 4526 scope.go:122] "RemoveContainer" containerID="462914b0592ca47d9212d3bde407fdbe8b0e0af3a4f34c79bd9ae474aadfa052" Oct 30 22:16:53.165865 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:16:53.165839 4526 scope.go:122] "RemoveContainer" containerID="721ac24cb1003d3548e484b970d6fdb5a44b4fd80011abdfdbe54d575ee8663d" Oct 30 22:17:12.476857 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:17:12.476814 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:17:20.049646 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:20.049608 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q"] Oct 30 22:17:20.098646 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:20.098614 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-ccf7q"] Oct 30 22:17:21.983657 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:21.983624 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fccba4-3698-413d-bd1a-11836e94bb20" path="/var/lib/kubelet/pods/93fccba4-3698-413d-bd1a-11836e94bb20/volumes" Oct 30 22:17:30.059516 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:30.050965 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm"] Oct 30 22:17:30.110515 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:30.095523 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wh4lm"] Oct 30 22:17:31.983727 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:31.983644 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740a7c8a-e79b-4a07-82bb-69da6f042164" path="/var/lib/kubelet/pods/740a7c8a-e79b-4a07-82bb-69da6f042164/volumes" Oct 30 22:17:35.808924 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.808894 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q"] Oct 30 22:17:35.809337 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.809263 4526 cpu_manager.go:414] "containerMap: removing container" podUID="9d52084f-c32f-4822-845e-15cdf2536113" containerName="revoke-keys" Oct 30 22:17:35.809337 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.809275 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d52084f-c32f-4822-845e-15cdf2536113" containerName="revoke-keys" Oct 30 22:17:35.809337 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.809332 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de8347c-98a0-47c0-aedb-496a981cddb3" containerName="cleanup" Oct 30 22:17:35.809467 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.809342 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d52084f-c32f-4822-845e-15cdf2536113" containerName="revoke-keys" Oct 30 22:17:35.812601 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.812578 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.818110 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.818085 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.818245 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.818129 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86w6\" (UniqueName: \"kubernetes.io/projected/82f7333e-164b-4750-b798-a2fd82479f9d-kube-api-access-t86w6\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.818303 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.818279 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.864476 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.864403 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:35.864697 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.864481 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:17:35.913676 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.913643 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:35.913676 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.913667 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:35.915598 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.915571 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q"] Oct 30 22:17:35.918828 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.918812 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.918902 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.918880 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.918968 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.918903 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86w6\" (UniqueName: \"kubernetes.io/projected/82f7333e-164b-4750-b798-a2fd82479f9d-kube-api-access-t86w6\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.919584 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.919566 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.921392 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.921375 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:35.976243 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:35.976205 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86w6\" (UniqueName: \"kubernetes.io/projected/82f7333e-164b-4750-b798-a2fd82479f9d-kube-api-access-t86w6\") pod \"maas-api-revoke-sub-d576e68b-flw5q\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:36.125014 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:36.124920 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:36.496232 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:36.496199 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q"] Oct 30 22:17:36.498533 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:17:36.498479 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f7333e_164b_4750_b798_a2fd82479f9d.slice/crio-3890b8b473fab10e9aea61936338046ba26762d57023a47ea1e1af2c3f131b89 WatchSource:0}: Error finding container 3890b8b473fab10e9aea61936338046ba26762d57023a47ea1e1af2c3f131b89: Status 404 returned error can't find the container with id 3890b8b473fab10e9aea61936338046ba26762d57023a47ea1e1af2c3f131b89 Oct 30 22:17:36.704648 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:36.704606 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" event={"ID":"82f7333e-164b-4750-b798-a2fd82479f9d","Type":"ContainerStarted","Data":"3890b8b473fab10e9aea61936338046ba26762d57023a47ea1e1af2c3f131b89"} Oct 30 22:17:37.710602 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:37.710518 4526 generic.go:356] "Generic (PLEG): container finished" podID="82f7333e-164b-4750-b798-a2fd82479f9d" containerID="5f510370eafc047facb9ba43cf261940155afa7213e91ae5de15f33b5024dc4d" exitCode=0 Oct 30 22:17:37.710980 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:37.710617 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" event={"ID":"82f7333e-164b-4750-b798-a2fd82479f9d","Type":"ContainerDied","Data":"5f510370eafc047facb9ba43cf261940155afa7213e91ae5de15f33b5024dc4d"} Oct 30 22:17:38.854762 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:38.854732 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:38.950831 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:38.950791 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-xks-ca\") pod \"82f7333e-164b-4750-b798-a2fd82479f9d\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " Oct 30 22:17:38.951043 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:38.950858 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f7333e-164b-4750-b798-a2fd82479f9d-kube-api-access-t86w6\" (UniqueName: \"kubernetes.io/projected/82f7333e-164b-4750-b798-a2fd82479f9d-kube-api-access-t86w6\") pod \"82f7333e-164b-4750-b798-a2fd82479f9d\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " Oct 30 22:17:38.951043 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:38.951000 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-openshift-ca\") pod \"82f7333e-164b-4750-b798-a2fd82479f9d\" (UID: \"82f7333e-164b-4750-b798-a2fd82479f9d\") " Oct 30 22:17:38.951330 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:38.951300 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-openshift-ca" pod "82f7333e-164b-4750-b798-a2fd82479f9d" (UID: "82f7333e-164b-4750-b798-a2fd82479f9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:38.951452 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:38.951436 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:38.953227 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:38.953198 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-xks-ca" pod "82f7333e-164b-4750-b798-a2fd82479f9d" (UID: "82f7333e-164b-4750-b798-a2fd82479f9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:38.953325 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:38.953262 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f7333e-164b-4750-b798-a2fd82479f9d-kube-api-access-t86w6" pod "82f7333e-164b-4750-b798-a2fd82479f9d" (UID: "82f7333e-164b-4750-b798-a2fd82479f9d"). InnerVolumeSpecName "kube-api-access-t86w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:39.052995 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:39.052951 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f7333e-164b-4750-b798-a2fd82479f9d-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:39.052995 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:39.052995 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86w6\" (UniqueName: \"kubernetes.io/projected/82f7333e-164b-4750-b798-a2fd82479f9d-kube-api-access-t86w6\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:39.724144 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:39.724113 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" Oct 30 22:17:39.724144 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:39.724132 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q" event={"ID":"82f7333e-164b-4750-b798-a2fd82479f9d","Type":"ContainerDied","Data":"3890b8b473fab10e9aea61936338046ba26762d57023a47ea1e1af2c3f131b89"} Oct 30 22:17:39.724381 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:39.724174 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3890b8b473fab10e9aea61936338046ba26762d57023a47ea1e1af2c3f131b89" Oct 30 22:17:42.477427 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:17:42.477395 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:17:53.186815 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:53.186782 4526 scope.go:122] "RemoveContainer" containerID="1827c1773dc01c20bb2e25efdd24cab0d09495a1c97ac0f6235588b93ca576b1" Oct 30 22:17:53.196436 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:17:53.196413 4526 scope.go:122] "RemoveContainer" containerID="183a637df12f20d31ede3c76e820d0b5b0e763ddc09c8e5040c263e03aecfbd8" Oct 30 22:18:12.477863 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:18:12.477816 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:18:14.861537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.861454 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9"] Oct 30 22:18:14.862079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.861917 4526 cpu_manager.go:414] "containerMap: removing container" podUID="82f7333e-164b-4750-b798-a2fd82479f9d" containerName="revoke-keys" Oct 30 22:18:14.862079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.861931 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f7333e-164b-4750-b798-a2fd82479f9d" containerName="revoke-keys" Oct 30 22:18:14.862079 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.862016 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f7333e-164b-4750-b798-a2fd82479f9d" containerName="revoke-keys" Oct 30 22:18:14.865244 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.865221 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:14.922170 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.922133 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9"] Oct 30 22:18:14.941548 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.941510 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bww7q\" (UniqueName: \"kubernetes.io/projected/a0f79ad0-cb71-480c-ad0a-4277fc901800-kube-api-access-bww7q\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:14.941717 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.941624 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:14.941717 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.941658 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:14.969093 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.969062 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:18:14.969265 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.969237 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:18:14.969332 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.969259 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:18:14.969332 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:14.969287 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:18:15.042856 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.042821 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:15.042856 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.042861 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:15.043123 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.042912 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bww7q\" (UniqueName: \"kubernetes.io/projected/a0f79ad0-cb71-480c-ad0a-4277fc901800-kube-api-access-bww7q\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:15.043653 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.043630 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:15.045335 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.045315 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:15.100551 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.100516 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bww7q\" (UniqueName: \"kubernetes.io/projected/a0f79ad0-cb71-480c-ad0a-4277fc901800-kube-api-access-bww7q\") pod \"maas-api-revoke-sub-d2bc93bc-hmtr9\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:15.178650 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.178563 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:15.548731 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.548704 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9"] Oct 30 22:18:15.922135 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.922095 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" event={"ID":"a0f79ad0-cb71-480c-ad0a-4277fc901800","Type":"ContainerStarted","Data":"51cc8c266cba3902f3ebf307f7e25f7e7565894249fb305e09687c14cef3cbd6"} Oct 30 22:18:15.922556 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:15.922151 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" event={"ID":"a0f79ad0-cb71-480c-ad0a-4277fc901800","Type":"ContainerStarted","Data":"988ee476804fca7ccf542441a12ba29934b170d51d1b523d647f80161d3e1606"} Oct 30 22:18:16.035826 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:16.035775 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" podStartSLOduration=2.035760993 podStartE2EDuration="2.035760993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:16.034918607 +0000 UTC m=+3686.168291806" watchObservedRunningTime="2026-10-30 22:18:16.035760993 +0000 UTC m=+3686.169134191" Oct 30 22:18:16.928276 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:16.928245 4526 generic.go:356] "Generic (PLEG): container finished" podID="a0f79ad0-cb71-480c-ad0a-4277fc901800" containerID="51cc8c266cba3902f3ebf307f7e25f7e7565894249fb305e09687c14cef3cbd6" exitCode=0 Oct 30 22:18:16.928718 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:16.928283 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" event={"ID":"a0f79ad0-cb71-480c-ad0a-4277fc901800","Type":"ContainerDied","Data":"51cc8c266cba3902f3ebf307f7e25f7e7565894249fb305e09687c14cef3cbd6"} Oct 30 22:18:18.070124 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.070098 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:18.173076 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.173044 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f79ad0-cb71-480c-ad0a-4277fc901800-kube-api-access-bww7q\" (UniqueName: \"kubernetes.io/projected/a0f79ad0-cb71-480c-ad0a-4277fc901800-kube-api-access-bww7q\") pod \"a0f79ad0-cb71-480c-ad0a-4277fc901800\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " Oct 30 22:18:18.173236 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.173117 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-openshift-ca\") pod \"a0f79ad0-cb71-480c-ad0a-4277fc901800\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " Oct 30 22:18:18.173291 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.173274 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-xks-ca\") pod \"a0f79ad0-cb71-480c-ad0a-4277fc901800\" (UID: \"a0f79ad0-cb71-480c-ad0a-4277fc901800\") " Oct 30 22:18:18.173420 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.173399 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-openshift-ca" pod "a0f79ad0-cb71-480c-ad0a-4277fc901800" (UID: "a0f79ad0-cb71-480c-ad0a-4277fc901800"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:18.173609 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.173595 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:18.175355 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.175326 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f79ad0-cb71-480c-ad0a-4277fc901800-kube-api-access-bww7q" pod "a0f79ad0-cb71-480c-ad0a-4277fc901800" (UID: "a0f79ad0-cb71-480c-ad0a-4277fc901800"). InnerVolumeSpecName "kube-api-access-bww7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:18.175449 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.175377 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-xks-ca" pod "a0f79ad0-cb71-480c-ad0a-4277fc901800" (UID: "a0f79ad0-cb71-480c-ad0a-4277fc901800"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:18.274144 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.274061 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f79ad0-cb71-480c-ad0a-4277fc901800-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:18.274144 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.274090 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bww7q\" (UniqueName: \"kubernetes.io/projected/a0f79ad0-cb71-480c-ad0a-4277fc901800-kube-api-access-bww7q\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:18.938326 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.938290 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" event={"ID":"a0f79ad0-cb71-480c-ad0a-4277fc901800","Type":"ContainerDied","Data":"988ee476804fca7ccf542441a12ba29934b170d51d1b523d647f80161d3e1606"} Oct 30 22:18:18.938326 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.938312 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9" Oct 30 22:18:18.938624 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:18.938339 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="988ee476804fca7ccf542441a12ba29934b170d51d1b523d647f80161d3e1606" Oct 30 22:18:42.477315 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:18:42.477278 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:18:53.220721 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:53.220691 4526 scope.go:122] "RemoveContainer" containerID="ed76c9831609c508fdb7c0f70b5877d8976492f2088e9e11b7ecf41de22d393a" Oct 30 22:18:53.229677 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:53.229659 4526 scope.go:122] "RemoveContainer" containerID="88abe9154087eab1d7ca37f3f652f74e8be23475990b888dd008fc35710f743b" Oct 30 22:18:53.239688 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:18:53.239664 4526 scope.go:122] "RemoveContainer" containerID="78dfdbdf6ea43ca3581c7b61ec4390adcac05972429612a3b8cd81036e30fbeb" Oct 30 22:19:12.477569 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:19:12.477528 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:19:14.658418 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.658375 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x"] Oct 30 22:19:14.659604 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.658879 4526 cpu_manager.go:414] "containerMap: removing container" podUID="a0f79ad0-cb71-480c-ad0a-4277fc901800" containerName="revoke-keys" Oct 30 22:19:14.659604 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.658911 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f79ad0-cb71-480c-ad0a-4277fc901800" containerName="revoke-keys" Oct 30 22:19:14.659604 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.658987 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f79ad0-cb71-480c-ad0a-4277fc901800" containerName="revoke-keys" Oct 30 22:19:14.662191 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.662169 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:14.765017 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.764984 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c60cfe\"" Oct 30 22:19:14.765210 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.765047 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7db25d04-dockercfg-tpmjl\"" Oct 30 22:19:14.767324 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.767301 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x"] Oct 30 22:19:14.842673 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.842635 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x\" (UID: \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:14.842872 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.842680 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45x6z\" (UniqueName: \"kubernetes.io/projected/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-kube-api-access-45x6z\") pod \"payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x\" (UID: \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:14.943589 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.943466 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x\" (UID: \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:14.943589 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.943561 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45x6z\" (UniqueName: \"kubernetes.io/projected/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-kube-api-access-45x6z\") pod \"payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x\" (UID: \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:14.944201 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:14.944178 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x\" (UID: \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:15.002327 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:15.002293 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45x6z\" (UniqueName: \"kubernetes.io/projected/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-kube-api-access-45x6z\") pod \"payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x\" (UID: \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:15.274994 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:15.274902 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:15.649575 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:15.649547 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x"] Oct 30 22:19:15.651874 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:19:15.651846 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cbdebd_1bb2_4672_9845_4d6a35d2ecf1.slice/crio-38996ae12037d2d71967897ad5354c5f6c1201bd6b1004dde06aea46593d4030 WatchSource:0}: Error finding container 38996ae12037d2d71967897ad5354c5f6c1201bd6b1004dde06aea46593d4030: Status 404 returned error can't find the container with id 38996ae12037d2d71967897ad5354c5f6c1201bd6b1004dde06aea46593d4030 Oct 30 22:19:16.194878 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:16.194841 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" event={"ID":"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1","Type":"ContainerStarted","Data":"e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6"} Oct 30 22:19:16.195329 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:16.194899 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" event={"ID":"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1","Type":"ContainerStarted","Data":"38996ae12037d2d71967897ad5354c5f6c1201bd6b1004dde06aea46593d4030"} Oct 30 22:19:16.195329 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:16.195013 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:16.312361 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:16.312303 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" podStartSLOduration=2.312289145 podStartE2EDuration="2.312289145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:16.309662279 +0000 UTC m=+3746.443035473" watchObservedRunningTime="2026-10-30 22:19:16.312289145 +0000 UTC m=+3746.445662342" Oct 30 22:19:27.202551 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:27.202516 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:19:42.477370 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:19:42.477326 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:19:58.076153 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:58.076113 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs"] Oct 30 22:19:58.125276 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:58.125233 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d802c957-cbncs"] Oct 30 22:19:59.983516 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:19:59.983454 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5da7764-04f5-45ad-910b-f367ec758e2d" path="/var/lib/kubelet/pods/a5da7764-04f5-45ad-910b-f367ec758e2d/volumes" Oct 30 22:20:12.477973 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:20:12.477935 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:20:39.046429 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:39.046334 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws"] Oct 30 22:20:39.095799 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:39.095763 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s7mws"] Oct 30 22:20:39.990422 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:39.990386 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d52084f-c32f-4822-845e-15cdf2536113" path="/var/lib/kubelet/pods/9d52084f-c32f-4822-845e-15cdf2536113/volumes" Oct 30 22:20:42.476784 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:20:42.476753 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:20:46.357447 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.357402 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9"] Oct 30 22:20:46.361346 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.361323 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.413883 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.413848 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:20:46.419353 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.419320 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.419821 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.419398 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjn62\" (UniqueName: \"kubernetes.io/projected/25171821-9a2a-434e-bf91-03c8ba9ab758-kube-api-access-vjn62\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.419821 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.419555 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.463082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.463045 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:20:46.463082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.463066 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:20:46.463082 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.463044 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:20:46.464894 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.464874 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9"] Oct 30 22:20:46.521022 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.520986 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.521236 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.521086 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjn62\" (UniqueName: \"kubernetes.io/projected/25171821-9a2a-434e-bf91-03c8ba9ab758-kube-api-access-vjn62\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.521341 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.521320 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.522019 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.521991 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.523746 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.523727 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.578085 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.578050 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjn62\" (UniqueName: \"kubernetes.io/projected/25171821-9a2a-434e-bf91-03c8ba9ab758-kube-api-access-vjn62\") pod \"maas-api-revoke-sub-2c905053-fsnb9\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:46.675469 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:46.675365 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:47.048036 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:47.048003 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9"] Oct 30 22:20:47.051144 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:20:47.051118 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25171821_9a2a_434e_bf91_03c8ba9ab758.slice/crio-468d5f3664b1b66c901b2a4244b1461deaa9b52ee4e18484ff98f324612e0dc3 WatchSource:0}: Error finding container 468d5f3664b1b66c901b2a4244b1461deaa9b52ee4e18484ff98f324612e0dc3: Status 404 returned error can't find the container with id 468d5f3664b1b66c901b2a4244b1461deaa9b52ee4e18484ff98f324612e0dc3 Oct 30 22:20:47.612641 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:47.612542 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" event={"ID":"25171821-9a2a-434e-bf91-03c8ba9ab758","Type":"ContainerStarted","Data":"dd6d92687043fa7efe4fd3fd2835fc11823121ba22163cc7ec5f0f9d1f13f7bb"} Oct 30 22:20:47.612641 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:47.612600 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" event={"ID":"25171821-9a2a-434e-bf91-03c8ba9ab758","Type":"ContainerStarted","Data":"468d5f3664b1b66c901b2a4244b1461deaa9b52ee4e18484ff98f324612e0dc3"} Oct 30 22:20:47.729731 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:47.729673 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" podStartSLOduration=1.7296568140000002 podStartE2EDuration="1.729656814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:47.728062287 +0000 UTC m=+3837.861435487" watchObservedRunningTime="2026-10-30 22:20:47.729656814 +0000 UTC m=+3837.863030034" Oct 30 22:20:48.618925 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:48.618889 4526 generic.go:356] "Generic (PLEG): container finished" podID="25171821-9a2a-434e-bf91-03c8ba9ab758" containerID="dd6d92687043fa7efe4fd3fd2835fc11823121ba22163cc7ec5f0f9d1f13f7bb" exitCode=0 Oct 30 22:20:48.619317 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:48.618971 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" event={"ID":"25171821-9a2a-434e-bf91-03c8ba9ab758","Type":"ContainerDied","Data":"dd6d92687043fa7efe4fd3fd2835fc11823121ba22163cc7ec5f0f9d1f13f7bb"} Oct 30 22:20:49.756528 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.756504 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:49.856758 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.856729 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-openshift-ca\") pod \"25171821-9a2a-434e-bf91-03c8ba9ab758\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " Oct 30 22:20:49.856952 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.856771 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-xks-ca\") pod \"25171821-9a2a-434e-bf91-03c8ba9ab758\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " Oct 30 22:20:49.856952 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.856821 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25171821-9a2a-434e-bf91-03c8ba9ab758-kube-api-access-vjn62\" (UniqueName: \"kubernetes.io/projected/25171821-9a2a-434e-bf91-03c8ba9ab758-kube-api-access-vjn62\") pod \"25171821-9a2a-434e-bf91-03c8ba9ab758\" (UID: \"25171821-9a2a-434e-bf91-03c8ba9ab758\") " Oct 30 22:20:49.857161 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.857133 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-openshift-ca" pod "25171821-9a2a-434e-bf91-03c8ba9ab758" (UID: "25171821-9a2a-434e-bf91-03c8ba9ab758"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:49.859140 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.859109 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25171821-9a2a-434e-bf91-03c8ba9ab758-kube-api-access-vjn62" pod "25171821-9a2a-434e-bf91-03c8ba9ab758" (UID: "25171821-9a2a-434e-bf91-03c8ba9ab758"). InnerVolumeSpecName "kube-api-access-vjn62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:49.859263 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.859158 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-xks-ca" pod "25171821-9a2a-434e-bf91-03c8ba9ab758" (UID: "25171821-9a2a-434e-bf91-03c8ba9ab758"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:49.958508 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.958409 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjn62\" (UniqueName: \"kubernetes.io/projected/25171821-9a2a-434e-bf91-03c8ba9ab758-kube-api-access-vjn62\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:49.958508 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.958439 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:49.958508 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:49.958449 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25171821-9a2a-434e-bf91-03c8ba9ab758-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:50.629977 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:50.629940 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" event={"ID":"25171821-9a2a-434e-bf91-03c8ba9ab758","Type":"ContainerDied","Data":"468d5f3664b1b66c901b2a4244b1461deaa9b52ee4e18484ff98f324612e0dc3"} Oct 30 22:20:50.630165 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:50.629987 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9" Oct 30 22:20:50.630165 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:50.629991 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="468d5f3664b1b66c901b2a4244b1461deaa9b52ee4e18484ff98f324612e0dc3" Oct 30 22:20:58.056016 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.055972 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z"] Oct 30 22:20:58.056899 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.056879 4526 cpu_manager.go:414] "containerMap: removing container" podUID="25171821-9a2a-434e-bf91-03c8ba9ab758" containerName="revoke-keys" Oct 30 22:20:58.057041 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.056901 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25171821-9a2a-434e-bf91-03c8ba9ab758" containerName="revoke-keys" Oct 30 22:20:58.058588 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.057133 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="25171821-9a2a-434e-bf91-03c8ba9ab758" containerName="revoke-keys" Oct 30 22:20:58.060897 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.060867 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.113345 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.113306 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:20:58.113549 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.113349 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:20:58.136797 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.136744 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.136999 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.136847 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.136999 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.136942 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75g8l\" (UniqueName: \"kubernetes.io/projected/4e4acd7d-8212-45ba-8139-de0a9c15697e-kube-api-access-75g8l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.162843 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.162812 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:20:58.162843 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.162825 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:20:58.164897 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.164871 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z"] Oct 30 22:20:58.237935 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.237878 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75g8l\" (UniqueName: \"kubernetes.io/projected/4e4acd7d-8212-45ba-8139-de0a9c15697e-kube-api-access-75g8l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.238159 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.238036 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.238159 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.238075 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.238837 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.238810 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.240767 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.240742 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.295033 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.294999 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75g8l\" (UniqueName: \"kubernetes.io/projected/4e4acd7d-8212-45ba-8139-de0a9c15697e-kube-api-access-75g8l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.375331 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.375242 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:20:58.753909 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:58.753830 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z"] Oct 30 22:20:58.756263 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:20:58.756227 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4acd7d_8212_45ba_8139_de0a9c15697e.slice/crio-d4be1889e4b500c07ca501bd9aba3c95f650f034929ec6d1ff3057561a346a9e WatchSource:0}: Error finding container d4be1889e4b500c07ca501bd9aba3c95f650f034929ec6d1ff3057561a346a9e: Status 404 returned error can't find the container with id d4be1889e4b500c07ca501bd9aba3c95f650f034929ec6d1ff3057561a346a9e Oct 30 22:20:59.672471 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:59.672441 4526 generic.go:356] "Generic (PLEG): container finished" podID="4e4acd7d-8212-45ba-8139-de0a9c15697e" containerID="01614c12a371090716d88382de091debd98f30b60ff38b1825fe0c9ee6e663e2" exitCode=0 Oct 30 22:20:59.672471 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:59.672482 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" event={"ID":"4e4acd7d-8212-45ba-8139-de0a9c15697e","Type":"ContainerDied","Data":"01614c12a371090716d88382de091debd98f30b60ff38b1825fe0c9ee6e663e2"} Oct 30 22:20:59.673046 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:20:59.672546 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" event={"ID":"4e4acd7d-8212-45ba-8139-de0a9c15697e","Type":"ContainerStarted","Data":"d4be1889e4b500c07ca501bd9aba3c95f650f034929ec6d1ff3057561a346a9e"} Oct 30 22:21:00.830694 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:00.830663 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:21:00.967827 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:00.967726 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4acd7d-8212-45ba-8139-de0a9c15697e-kube-api-access-75g8l\" (UniqueName: \"kubernetes.io/projected/4e4acd7d-8212-45ba-8139-de0a9c15697e-kube-api-access-75g8l\") pod \"4e4acd7d-8212-45ba-8139-de0a9c15697e\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " Oct 30 22:21:00.967827 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:00.967804 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-openshift-ca\") pod \"4e4acd7d-8212-45ba-8139-de0a9c15697e\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " Oct 30 22:21:00.968088 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:00.967860 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-xks-ca\") pod \"4e4acd7d-8212-45ba-8139-de0a9c15697e\" (UID: \"4e4acd7d-8212-45ba-8139-de0a9c15697e\") " Oct 30 22:21:00.968212 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:00.968185 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-openshift-ca" pod "4e4acd7d-8212-45ba-8139-de0a9c15697e" (UID: "4e4acd7d-8212-45ba-8139-de0a9c15697e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:00.970112 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:00.970084 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-xks-ca" pod "4e4acd7d-8212-45ba-8139-de0a9c15697e" (UID: "4e4acd7d-8212-45ba-8139-de0a9c15697e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:00.970257 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:00.970213 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4acd7d-8212-45ba-8139-de0a9c15697e-kube-api-access-75g8l" pod "4e4acd7d-8212-45ba-8139-de0a9c15697e" (UID: "4e4acd7d-8212-45ba-8139-de0a9c15697e"). InnerVolumeSpecName "kube-api-access-75g8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:01.069752 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:01.069712 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75g8l\" (UniqueName: \"kubernetes.io/projected/4e4acd7d-8212-45ba-8139-de0a9c15697e-kube-api-access-75g8l\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:01.069752 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:01.069743 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:01.069752 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:01.069753 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4acd7d-8212-45ba-8139-de0a9c15697e-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:01.683510 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:01.683458 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" Oct 30 22:21:01.683728 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:01.683459 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z" event={"ID":"4e4acd7d-8212-45ba-8139-de0a9c15697e","Type":"ContainerDied","Data":"d4be1889e4b500c07ca501bd9aba3c95f650f034929ec6d1ff3057561a346a9e"} Oct 30 22:21:01.683728 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:01.683588 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4be1889e4b500c07ca501bd9aba3c95f650f034929ec6d1ff3057561a346a9e" Oct 30 22:21:09.541020 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:09.540980 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x"] Oct 30 22:21:12.478109 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:21:12.478071 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:21:14.541510 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:14.541459 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" podUID="94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" containerName="payload-processing" containerID="cri-o://e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6" gracePeriod=25 Oct 30 22:21:16.246589 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:16.246554 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z"] Oct 30 22:21:16.295161 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:16.295128 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c60cfe-g8m4z"] Oct 30 22:21:17.201253 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:17.201220 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" podUID="94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:21:17.985467 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:17.985427 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4acd7d-8212-45ba-8139-de0a9c15697e" path="/var/lib/kubelet/pods/4e4acd7d-8212-45ba-8139-de0a9c15697e/volumes" Oct 30 22:21:27.201542 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:27.201505 4526 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" podUID="94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:21:34.841761 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.841730 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:21:34.843979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.843943 4526 generic.go:356] "Generic (PLEG): container finished" podID="94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" containerID="e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6" exitCode=0 Oct 30 22:21:34.843979 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.843979 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" Oct 30 22:21:34.844215 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.844047 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" event={"ID":"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1","Type":"ContainerDied","Data":"e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6"} Oct 30 22:21:34.844215 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.844122 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x" event={"ID":"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1","Type":"ContainerDied","Data":"38996ae12037d2d71967897ad5354c5f6c1201bd6b1004dde06aea46593d4030"} Oct 30 22:21:34.844215 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.844160 4526 scope.go:122] "RemoveContainer" containerID="e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6" Oct 30 22:21:34.855325 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.855291 4526 scope.go:122] "RemoveContainer" containerID="e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6" Oct 30 22:21:34.855793 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:21:34.855752 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6\": container with ID starting with e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6 not found: ID does not exist" containerID="e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6" Oct 30 22:21:34.855892 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.855808 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6"} err="failed to get container status \"e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6\": rpc error: code = NotFound desc = could not find container \"e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6\": container with ID starting with e97fadc48b82e36d6c0d3545dbb163c141826d6a4d77c284ab350983db2853d6 not found: ID does not exist" Oct 30 22:21:34.903091 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.903033 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-plugins-config-volume\") pod \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\" (UID: \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\") " Oct 30 22:21:34.903314 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.903171 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-kube-api-access-45x6z\" (UniqueName: \"kubernetes.io/projected/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-kube-api-access-45x6z\") pod \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\" (UID: \"94cbdebd-1bb2-4672-9845-4d6a35d2ecf1\") " Oct 30 22:21:34.903469 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.903440 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-plugins-config-volume" pod "94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" (UID: "94cbdebd-1bb2-4672-9845-4d6a35d2ecf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:34.903582 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.903563 4526 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-plugins-config-volume\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:34.905588 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:34.905558 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-kube-api-access-45x6z" pod "94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" (UID: "94cbdebd-1bb2-4672-9845-4d6a35d2ecf1"). InnerVolumeSpecName "kube-api-access-45x6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:35.004722 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:35.004677 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45x6z\" (UniqueName: \"kubernetes.io/projected/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1-kube-api-access-45x6z\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:35.269801 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:35.269763 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x"] Oct 30 22:21:35.273999 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:35.273963 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c60cfe-997d987d9gd65x"] Oct 30 22:21:35.983671 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:35.983625 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" path="/var/lib/kubelet/pods/94cbdebd-1bb2-4672-9845-4d6a35d2ecf1/volumes" Oct 30 22:21:42.477548 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:21:42.477511 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:21:53.281589 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:53.281555 4526 scope.go:122] "RemoveContainer" containerID="678efbdf095f187b9bab256a9588d0ff508845a62ed83fa11b1d2ce996a6e5ce" Oct 30 22:21:53.289705 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:53.289677 4526 scope.go:122] "RemoveContainer" containerID="1d3441fd2f65e0e4387e5c4dd1f4e502391c6fa58307200104fb773004711299" Oct 30 22:21:53.297429 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:21:53.297407 4526 scope.go:122] "RemoveContainer" containerID="161b2b980a83331553f651ec919057d73a201cc34d4d9afa7bbcfb452973a750" Oct 30 22:22:12.479979 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:22:12.479941 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:22:39.051944 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:39.051905 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q"] Oct 30 22:22:39.101100 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:39.101061 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-flw5q"] Oct 30 22:22:39.983876 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:39.983838 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f7333e-164b-4750-b798-a2fd82479f9d" path="/var/lib/kubelet/pods/82f7333e-164b-4750-b798-a2fd82479f9d/volumes" Oct 30 22:22:42.477384 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:22:42.477339 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:22:46.813325 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.813286 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr"] Oct 30 22:22:46.813908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.813815 4526 cpu_manager.go:414] "containerMap: removing container" podUID="94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" containerName="payload-processing" Oct 30 22:22:46.813908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.813834 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" containerName="payload-processing" Oct 30 22:22:46.813908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.813871 4526 cpu_manager.go:414] "containerMap: removing container" podUID="4e4acd7d-8212-45ba-8139-de0a9c15697e" containerName="revoke-keys" Oct 30 22:22:46.813908 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.813878 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4acd7d-8212-45ba-8139-de0a9c15697e" containerName="revoke-keys" Oct 30 22:22:46.814148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.813985 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4acd7d-8212-45ba-8139-de0a9c15697e" containerName="revoke-keys" Oct 30 22:22:46.814148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.814005 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="94cbdebd-1bb2-4672-9845-4d6a35d2ecf1" containerName="payload-processing" Oct 30 22:22:46.817269 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.817249 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:46.874246 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.874214 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr"] Oct 30 22:22:46.921284 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.921255 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:22:46.921473 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.921452 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:22:46.921576 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.921467 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:22:46.921576 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.921531 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:22:46.950648 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.950616 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:46.950789 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.950753 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dk5q\" (UniqueName: \"kubernetes.io/projected/c96603be-4df0-45ca-98b2-92f599832c2c-kube-api-access-2dk5q\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:46.950789 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:46.950785 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:47.051465 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:47.051429 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dk5q\" (UniqueName: \"kubernetes.io/projected/c96603be-4df0-45ca-98b2-92f599832c2c-kube-api-access-2dk5q\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:47.051465 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:47.051468 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:47.051716 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:47.051571 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:47.052165 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:47.052144 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:47.054174 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:47.054151 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:47.108217 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:47.108148 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dk5q\" (UniqueName: \"kubernetes.io/projected/c96603be-4df0-45ca-98b2-92f599832c2c-kube-api-access-2dk5q\") pod \"maas-api-revoke-sub-452b6d7b-llvmr\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:47.129630 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:47.129599 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:47.502768 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:47.502735 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr"] Oct 30 22:22:47.505440 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:22:47.505407 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96603be_4df0_45ca_98b2_92f599832c2c.slice/crio-1a04d4d0645f6ef8a44b1f8a7f5899fd5c6804b7e02128d187c9fe2760ede689 WatchSource:0}: Error finding container 1a04d4d0645f6ef8a44b1f8a7f5899fd5c6804b7e02128d187c9fe2760ede689: Status 404 returned error can't find the container with id 1a04d4d0645f6ef8a44b1f8a7f5899fd5c6804b7e02128d187c9fe2760ede689 Oct 30 22:22:48.223636 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:48.223543 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" event={"ID":"c96603be-4df0-45ca-98b2-92f599832c2c","Type":"ContainerStarted","Data":"18084011b514ee6b6f8123bb2e67ef757ad01d57a59ead8fcd2a255961c83df3"} Oct 30 22:22:48.223636 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:48.223604 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" event={"ID":"c96603be-4df0-45ca-98b2-92f599832c2c","Type":"ContainerStarted","Data":"1a04d4d0645f6ef8a44b1f8a7f5899fd5c6804b7e02128d187c9fe2760ede689"} Oct 30 22:22:48.336668 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:48.336607 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" podStartSLOduration=2.3365907 podStartE2EDuration="2.3365907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:48.335351714 +0000 UTC m=+3958.468724913" watchObservedRunningTime="2026-10-30 22:22:48.3365907 +0000 UTC m=+3958.469963897" Oct 30 22:22:49.232333 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:49.232296 4526 generic.go:356] "Generic (PLEG): container finished" podID="c96603be-4df0-45ca-98b2-92f599832c2c" containerID="18084011b514ee6b6f8123bb2e67ef757ad01d57a59ead8fcd2a255961c83df3" exitCode=0 Oct 30 22:22:49.232900 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:49.232414 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" event={"ID":"c96603be-4df0-45ca-98b2-92f599832c2c","Type":"ContainerDied","Data":"18084011b514ee6b6f8123bb2e67ef757ad01d57a59ead8fcd2a255961c83df3"} Oct 30 22:22:50.669360 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.669111 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:22:50.683418 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.683378 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-openshift-ca\") pod \"c96603be-4df0-45ca-98b2-92f599832c2c\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " Oct 30 22:22:50.683611 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.683437 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-xks-ca\") pod \"c96603be-4df0-45ca-98b2-92f599832c2c\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " Oct 30 22:22:50.683671 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.683620 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96603be-4df0-45ca-98b2-92f599832c2c-kube-api-access-2dk5q\" (UniqueName: \"kubernetes.io/projected/c96603be-4df0-45ca-98b2-92f599832c2c-kube-api-access-2dk5q\") pod \"c96603be-4df0-45ca-98b2-92f599832c2c\" (UID: \"c96603be-4df0-45ca-98b2-92f599832c2c\") " Oct 30 22:22:50.683852 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.683827 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-openshift-ca" pod "c96603be-4df0-45ca-98b2-92f599832c2c" (UID: "c96603be-4df0-45ca-98b2-92f599832c2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:50.683944 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.683926 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:50.685833 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.685793 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96603be-4df0-45ca-98b2-92f599832c2c-kube-api-access-2dk5q" pod "c96603be-4df0-45ca-98b2-92f599832c2c" (UID: "c96603be-4df0-45ca-98b2-92f599832c2c"). InnerVolumeSpecName "kube-api-access-2dk5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:50.685833 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.685805 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-xks-ca" pod "c96603be-4df0-45ca-98b2-92f599832c2c" (UID: "c96603be-4df0-45ca-98b2-92f599832c2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:50.784449 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.784412 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dk5q\" (UniqueName: \"kubernetes.io/projected/c96603be-4df0-45ca-98b2-92f599832c2c-kube-api-access-2dk5q\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:50.784449 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:50.784443 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c96603be-4df0-45ca-98b2-92f599832c2c-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:51.271514 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:51.268071 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" event={"ID":"c96603be-4df0-45ca-98b2-92f599832c2c","Type":"ContainerDied","Data":"1a04d4d0645f6ef8a44b1f8a7f5899fd5c6804b7e02128d187c9fe2760ede689"} Oct 30 22:22:51.271514 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:51.268136 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a04d4d0645f6ef8a44b1f8a7f5899fd5c6804b7e02128d187c9fe2760ede689" Oct 30 22:22:51.271514 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:22:51.268202 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr" Oct 30 22:23:12.477879 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:23:12.477842 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:23:19.044940 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:19.044898 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9"] Oct 30 22:23:19.094179 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:19.094144 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hmtr9"] Oct 30 22:23:19.984120 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:19.984077 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f79ad0-cb71-480c-ad0a-4277fc901800" path="/var/lib/kubelet/pods/a0f79ad0-cb71-480c-ad0a-4277fc901800/volumes" Oct 30 22:23:30.127664 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.127626 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm"] Oct 30 22:23:30.128102 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.128001 4526 cpu_manager.go:414] "containerMap: removing container" podUID="c96603be-4df0-45ca-98b2-92f599832c2c" containerName="revoke-keys" Oct 30 22:23:30.128102 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.128012 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96603be-4df0-45ca-98b2-92f599832c2c" containerName="revoke-keys" Oct 30 22:23:30.128102 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.128079 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96603be-4df0-45ca-98b2-92f599832c2c" containerName="revoke-keys" Oct 30 22:23:30.131132 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.131116 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.183107 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.183069 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:30.209480 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.209442 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.209695 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.209527 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.209695 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.209603 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwgs\" (UniqueName: \"kubernetes.io/projected/083dad9d-22bc-424c-b89e-2eb451d75415-kube-api-access-bvwgs\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.232381 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.232346 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:30.232381 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.232358 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:30.232624 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.232561 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:23:30.234432 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.234410 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm"] Oct 30 22:23:30.310618 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.310579 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.310817 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.310631 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.310817 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.310740 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwgs\" (UniqueName: \"kubernetes.io/projected/083dad9d-22bc-424c-b89e-2eb451d75415-kube-api-access-bvwgs\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.311228 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.311209 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.313090 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.313073 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.372345 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.372315 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwgs\" (UniqueName: \"kubernetes.io/projected/083dad9d-22bc-424c-b89e-2eb451d75415-kube-api-access-bvwgs\") pod \"maas-api-revoke-sub-3e31a506-gdtkm\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.443321 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.443245 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:30.816852 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:30.816821 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm"] Oct 30 22:23:30.819750 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:23:30.819710 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083dad9d_22bc_424c_b89e_2eb451d75415.slice/crio-44e4c7f8066b4bbe53dcd7cf27a58af77ea2d02e8220f66af70e368f7e8ca279 WatchSource:0}: Error finding container 44e4c7f8066b4bbe53dcd7cf27a58af77ea2d02e8220f66af70e368f7e8ca279: Status 404 returned error can't find the container with id 44e4c7f8066b4bbe53dcd7cf27a58af77ea2d02e8220f66af70e368f7e8ca279 Oct 30 22:23:31.463231 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:31.463111 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" event={"ID":"083dad9d-22bc-424c-b89e-2eb451d75415","Type":"ContainerStarted","Data":"5340ebc54aba5258fa957c02fc753684e2c8b21e61f96a26c7642bc17d3564a3"} Oct 30 22:23:31.463231 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:31.463178 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" event={"ID":"083dad9d-22bc-424c-b89e-2eb451d75415","Type":"ContainerStarted","Data":"44e4c7f8066b4bbe53dcd7cf27a58af77ea2d02e8220f66af70e368f7e8ca279"} Oct 30 22:23:32.473455 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:32.473412 4526 generic.go:356] "Generic (PLEG): container finished" podID="083dad9d-22bc-424c-b89e-2eb451d75415" containerID="5340ebc54aba5258fa957c02fc753684e2c8b21e61f96a26c7642bc17d3564a3" exitCode=0 Oct 30 22:23:32.473894 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:32.473512 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" event={"ID":"083dad9d-22bc-424c-b89e-2eb451d75415","Type":"ContainerDied","Data":"5340ebc54aba5258fa957c02fc753684e2c8b21e61f96a26c7642bc17d3564a3"} Oct 30 22:23:33.615135 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.615113 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:33.674302 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.674271 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-xks-ca\") pod \"083dad9d-22bc-424c-b89e-2eb451d75415\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " Oct 30 22:23:33.674474 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.674331 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-openshift-ca\") pod \"083dad9d-22bc-424c-b89e-2eb451d75415\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " Oct 30 22:23:33.674474 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.674413 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083dad9d-22bc-424c-b89e-2eb451d75415-kube-api-access-bvwgs\" (UniqueName: \"kubernetes.io/projected/083dad9d-22bc-424c-b89e-2eb451d75415-kube-api-access-bvwgs\") pod \"083dad9d-22bc-424c-b89e-2eb451d75415\" (UID: \"083dad9d-22bc-424c-b89e-2eb451d75415\") " Oct 30 22:23:33.674729 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.674702 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-openshift-ca" pod "083dad9d-22bc-424c-b89e-2eb451d75415" (UID: "083dad9d-22bc-424c-b89e-2eb451d75415"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:33.676741 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.676711 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-xks-ca" pod "083dad9d-22bc-424c-b89e-2eb451d75415" (UID: "083dad9d-22bc-424c-b89e-2eb451d75415"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:33.676856 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.676757 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083dad9d-22bc-424c-b89e-2eb451d75415-kube-api-access-bvwgs" pod "083dad9d-22bc-424c-b89e-2eb451d75415" (UID: "083dad9d-22bc-424c-b89e-2eb451d75415"). InnerVolumeSpecName "kube-api-access-bvwgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:33.775557 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.775452 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:33.775557 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.775483 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvwgs\" (UniqueName: \"kubernetes.io/projected/083dad9d-22bc-424c-b89e-2eb451d75415-kube-api-access-bvwgs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:33.775557 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:33.775525 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083dad9d-22bc-424c-b89e-2eb451d75415-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:34.483392 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:34.483361 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" Oct 30 22:23:34.483631 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:34.483398 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm" event={"ID":"083dad9d-22bc-424c-b89e-2eb451d75415","Type":"ContainerDied","Data":"44e4c7f8066b4bbe53dcd7cf27a58af77ea2d02e8220f66af70e368f7e8ca279"} Oct 30 22:23:34.483631 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:34.483471 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44e4c7f8066b4bbe53dcd7cf27a58af77ea2d02e8220f66af70e368f7e8ca279" Oct 30 22:23:42.477203 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:23:42.477164 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:23:53.334698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:23:53.334664 4526 scope.go:122] "RemoveContainer" containerID="5f510370eafc047facb9ba43cf261940155afa7213e91ae5de15f33b5024dc4d" Oct 30 22:24:02.977048 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:02.977005 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq"] Oct 30 22:24:02.977510 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:02.977417 4526 cpu_manager.go:414] "containerMap: removing container" podUID="083dad9d-22bc-424c-b89e-2eb451d75415" containerName="revoke-keys" Oct 30 22:24:02.977510 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:02.977429 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083dad9d-22bc-424c-b89e-2eb451d75415" containerName="revoke-keys" Oct 30 22:24:02.977605 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:02.977512 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="083dad9d-22bc-424c-b89e-2eb451d75415" containerName="revoke-keys" Oct 30 22:24:02.980673 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:02.980640 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.082368 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.082329 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.082586 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.082455 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jfn\" (UniqueName: \"kubernetes.io/projected/fa19f503-b85f-472d-b4aa-0de0c2279e3e-kube-api-access-l2jfn\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.082586 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.082483 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.084160 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.084136 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:24:03.084250 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.084204 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:03.084250 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.084239 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:03.084390 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.084240 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:03.086116 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.086098 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq"] Oct 30 22:24:03.183658 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.183623 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jfn\" (UniqueName: \"kubernetes.io/projected/fa19f503-b85f-472d-b4aa-0de0c2279e3e-kube-api-access-l2jfn\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.183883 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.183667 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.183883 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.183714 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.184509 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.184408 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.186255 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.186229 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.240440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.240363 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jfn\" (UniqueName: \"kubernetes.io/projected/fa19f503-b85f-472d-b4aa-0de0c2279e3e-kube-api-access-l2jfn\") pod \"maas-api-revoke-sub-6fe33fe5-q5dnq\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.294922 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.294885 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:03.667805 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:03.667677 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq"] Oct 30 22:24:03.670153 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:24:03.670128 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa19f503_b85f_472d_b4aa_0de0c2279e3e.slice/crio-3e08664e1144b793e920d0dfab8e2b73585bc47a6f901ddc1fe0b2e8fe7b82d6 WatchSource:0}: Error finding container 3e08664e1144b793e920d0dfab8e2b73585bc47a6f901ddc1fe0b2e8fe7b82d6: Status 404 returned error can't find the container with id 3e08664e1144b793e920d0dfab8e2b73585bc47a6f901ddc1fe0b2e8fe7b82d6 Oct 30 22:24:04.654122 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:04.654088 4526 generic.go:356] "Generic (PLEG): container finished" podID="fa19f503-b85f-472d-b4aa-0de0c2279e3e" containerID="c5699f96f5faef1a5c9615e0674e349b106375a88af92f3f7f31b18b9e778452" exitCode=0 Oct 30 22:24:04.654531 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:04.654237 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" event={"ID":"fa19f503-b85f-472d-b4aa-0de0c2279e3e","Type":"ContainerDied","Data":"c5699f96f5faef1a5c9615e0674e349b106375a88af92f3f7f31b18b9e778452"} Oct 30 22:24:04.654531 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:04.654286 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" event={"ID":"fa19f503-b85f-472d-b4aa-0de0c2279e3e","Type":"ContainerStarted","Data":"3e08664e1144b793e920d0dfab8e2b73585bc47a6f901ddc1fe0b2e8fe7b82d6"} Oct 30 22:24:05.803966 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:05.803940 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:05.908814 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:05.908779 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa19f503-b85f-472d-b4aa-0de0c2279e3e-kube-api-access-l2jfn\" (UniqueName: \"kubernetes.io/projected/fa19f503-b85f-472d-b4aa-0de0c2279e3e-kube-api-access-l2jfn\") pod \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " Oct 30 22:24:05.908814 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:05.908817 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-openshift-ca\") pod \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " Oct 30 22:24:05.909054 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:05.908845 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-xks-ca\") pod \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\" (UID: \"fa19f503-b85f-472d-b4aa-0de0c2279e3e\") " Oct 30 22:24:05.909251 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:05.909223 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-openshift-ca" pod "fa19f503-b85f-472d-b4aa-0de0c2279e3e" (UID: "fa19f503-b85f-472d-b4aa-0de0c2279e3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:05.911216 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:05.911163 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-xks-ca" pod "fa19f503-b85f-472d-b4aa-0de0c2279e3e" (UID: "fa19f503-b85f-472d-b4aa-0de0c2279e3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:05.911216 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:05.911171 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa19f503-b85f-472d-b4aa-0de0c2279e3e-kube-api-access-l2jfn" pod "fa19f503-b85f-472d-b4aa-0de0c2279e3e" (UID: "fa19f503-b85f-472d-b4aa-0de0c2279e3e"). InnerVolumeSpecName "kube-api-access-l2jfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:06.009719 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:06.009683 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2jfn\" (UniqueName: \"kubernetes.io/projected/fa19f503-b85f-472d-b4aa-0de0c2279e3e-kube-api-access-l2jfn\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:06.009719 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:06.009715 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-openshift-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:06.009719 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:06.009725 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa19f503-b85f-472d-b4aa-0de0c2279e3e-maas-api-xks-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:06.672356 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:06.672321 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" Oct 30 22:24:06.672586 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:06.672346 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq" event={"ID":"fa19f503-b85f-472d-b4aa-0de0c2279e3e","Type":"ContainerDied","Data":"3e08664e1144b793e920d0dfab8e2b73585bc47a6f901ddc1fe0b2e8fe7b82d6"} Oct 30 22:24:06.672586 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:06.672432 4526 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e08664e1144b793e920d0dfab8e2b73585bc47a6f901ddc1fe0b2e8fe7b82d6" Oct 30 22:24:12.477829 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:24:12.477796 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:24:26.227563 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.227526 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-2tnr9"] Oct 30 22:24:26.228154 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.228078 4526 cpu_manager.go:414] "containerMap: removing container" podUID="fa19f503-b85f-472d-b4aa-0de0c2279e3e" containerName="revoke-keys" Oct 30 22:24:26.228154 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.228095 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa19f503-b85f-472d-b4aa-0de0c2279e3e" containerName="revoke-keys" Oct 30 22:24:26.228284 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.228159 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa19f503-b85f-472d-b4aa-0de0c2279e3e" containerName="revoke-keys" Oct 30 22:24:26.231379 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.231364 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.309463 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.309425 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-maas-api-tls\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.309661 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.309502 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-metrics-certs\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.309708 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.309647 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdw7z\" (UniqueName: \"kubernetes.io/projected/37e07277-b637-4e6f-b161-793af306f3ed-kube-api-access-rdw7z\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.309806 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.309783 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e07277-b637-4e6f-b161-793af306f3ed-openshift-service-ca\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.332917 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.332878 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:26.332917 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.332907 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:26.332917 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.332921 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk7w4\"" Oct 30 22:24:26.332917 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.332923 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:24:26.333415 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.332981 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 22:24:26.384216 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.384182 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-2tnr9"] Oct 30 22:24:26.411115 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.411084 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-maas-api-tls\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.411334 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.411145 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-metrics-certs\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.411334 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.411184 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdw7z\" (UniqueName: \"kubernetes.io/projected/37e07277-b637-4e6f-b161-793af306f3ed-kube-api-access-rdw7z\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.411334 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.411255 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e07277-b637-4e6f-b161-793af306f3ed-openshift-service-ca\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.411943 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.411919 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e07277-b637-4e6f-b161-793af306f3ed-openshift-service-ca\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.413848 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.413829 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-metrics-certs\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.413975 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.413953 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-maas-api-tls\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.469516 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.469455 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdw7z\" (UniqueName: \"kubernetes.io/projected/37e07277-b637-4e6f-b161-793af306f3ed-kube-api-access-rdw7z\") pod \"maas-api-745b76d875-2tnr9\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.545262 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.545229 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:26.918791 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.918767 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-2tnr9"] Oct 30 22:24:26.921318 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:24:26.921290 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e07277_b637_4e6f_b161_793af306f3ed.slice/crio-a9afd7974fd7d9d534a6939a4a2a3f01e86f8bcd3a7b0aaa0e11ff3e9ae6093f WatchSource:0}: Error finding container a9afd7974fd7d9d534a6939a4a2a3f01e86f8bcd3a7b0aaa0e11ff3e9ae6093f: Status 404 returned error can't find the container with id a9afd7974fd7d9d534a6939a4a2a3f01e86f8bcd3a7b0aaa0e11ff3e9ae6093f Oct 30 22:24:26.922656 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:26.922634 4526 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:24:27.774815 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:27.774781 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" event={"ID":"37e07277-b637-4e6f-b161-793af306f3ed","Type":"ContainerStarted","Data":"a9afd7974fd7d9d534a6939a4a2a3f01e86f8bcd3a7b0aaa0e11ff3e9ae6093f"} Oct 30 22:24:28.781607 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:28.781567 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" event={"ID":"37e07277-b637-4e6f-b161-793af306f3ed","Type":"ContainerStarted","Data":"c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6"} Oct 30 22:24:28.782028 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:28.781750 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:28.895694 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:28.895633 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" podStartSLOduration=2.002162012 podStartE2EDuration="2.895614439s" totalImagesPullingTime="893.452427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:28.894984384 +0000 UTC m=+4059.028357593" watchObservedRunningTime="2026-10-30 22:24:28.895614439 +0000 UTC m=+4059.028987640" Oct 30 22:24:34.792375 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.792345 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:34.830754 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.830723 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-2tnr9"] Oct 30 22:24:34.830948 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.830926 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" podUID="37e07277-b637-4e6f-b161-793af306f3ed" containerName="maas-api" containerID="cri-o://c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6" gracePeriod=30 Oct 30 22:24:34.880801 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.880768 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5644488b47-r9zp9"] Oct 30 22:24:34.884222 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.884205 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:34.928785 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.928749 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5644488b47-r9zp9"] Oct 30 22:24:34.990178 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.990143 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmq2\" (UniqueName: \"kubernetes.io/projected/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-kube-api-access-6tmq2\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:34.990178 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.990181 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-openshift-service-ca\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:34.990467 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.990212 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-maas-api-tls\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:34.990467 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:34.990325 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-metrics-certs\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.087383 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.087354 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:35.091589 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.091568 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-maas-api-tls\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.091721 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.091614 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-metrics-certs\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.091794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.091743 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmq2\" (UniqueName: \"kubernetes.io/projected/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-kube-api-access-6tmq2\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.091794 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.091774 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-openshift-service-ca\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.092337 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.092317 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-openshift-service-ca\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.094092 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.094074 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-metrics-certs\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.094213 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.094195 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-maas-api-tls\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.147980 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.147946 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmq2\" (UniqueName: \"kubernetes.io/projected/05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7-kube-api-access-6tmq2\") pod \"maas-api-5644488b47-r9zp9\" (UID: \"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.192890 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.192858 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e07277-b637-4e6f-b161-793af306f3ed-kube-api-access-rdw7z\" (UniqueName: \"kubernetes.io/projected/37e07277-b637-4e6f-b161-793af306f3ed-kube-api-access-rdw7z\") pod \"37e07277-b637-4e6f-b161-793af306f3ed\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " Oct 30 22:24:35.193050 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.192906 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-metrics-certs\") pod \"37e07277-b637-4e6f-b161-793af306f3ed\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " Oct 30 22:24:35.193050 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.192979 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-maas-api-tls\") pod \"37e07277-b637-4e6f-b161-793af306f3ed\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " Oct 30 22:24:35.193050 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.193035 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e07277-b637-4e6f-b161-793af306f3ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e07277-b637-4e6f-b161-793af306f3ed-openshift-service-ca\") pod \"37e07277-b637-4e6f-b161-793af306f3ed\" (UID: \"37e07277-b637-4e6f-b161-793af306f3ed\") " Oct 30 22:24:35.193440 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.193412 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e07277-b637-4e6f-b161-793af306f3ed-openshift-service-ca" pod "37e07277-b637-4e6f-b161-793af306f3ed" (UID: "37e07277-b637-4e6f-b161-793af306f3ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:35.195165 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.195142 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-maas-api-tls" pod "37e07277-b637-4e6f-b161-793af306f3ed" (UID: "37e07277-b637-4e6f-b161-793af306f3ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:35.195271 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.195189 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-metrics-certs" pod "37e07277-b637-4e6f-b161-793af306f3ed" (UID: "37e07277-b637-4e6f-b161-793af306f3ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:35.195271 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.195209 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e07277-b637-4e6f-b161-793af306f3ed-kube-api-access-rdw7z" pod "37e07277-b637-4e6f-b161-793af306f3ed" (UID: "37e07277-b637-4e6f-b161-793af306f3ed"). InnerVolumeSpecName "kube-api-access-rdw7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:35.220243 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.220219 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:35.295071 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.295040 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdw7z\" (UniqueName: \"kubernetes.io/projected/37e07277-b637-4e6f-b161-793af306f3ed-kube-api-access-rdw7z\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:35.295215 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.295078 4526 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-metrics-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:35.295215 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.295098 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e07277-b637-4e6f-b161-793af306f3ed-maas-api-tls\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:35.295215 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.295114 4526 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e07277-b637-4e6f-b161-793af306f3ed-openshift-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:35.592022 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.592000 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5644488b47-r9zp9"] Oct 30 22:24:35.594321 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:24:35.594296 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aa8ae1_8ebf_40eb_b8ba_dbb05b1f4ed7.slice/crio-963542bce52a266bd177a7f51783ac0807c03cf1c9189981ff971672e017ac83 WatchSource:0}: Error finding container 963542bce52a266bd177a7f51783ac0807c03cf1c9189981ff971672e017ac83: Status 404 returned error can't find the container with id 963542bce52a266bd177a7f51783ac0807c03cf1c9189981ff971672e017ac83 Oct 30 22:24:35.819307 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.819269 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" event={"ID":"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7","Type":"ContainerStarted","Data":"963542bce52a266bd177a7f51783ac0807c03cf1c9189981ff971672e017ac83"} Oct 30 22:24:35.820565 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.820545 4526 generic.go:356] "Generic (PLEG): container finished" podID="37e07277-b637-4e6f-b161-793af306f3ed" containerID="c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6" exitCode=0 Oct 30 22:24:35.820666 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.820590 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" Oct 30 22:24:35.820666 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.820614 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" event={"ID":"37e07277-b637-4e6f-b161-793af306f3ed","Type":"ContainerDied","Data":"c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6"} Oct 30 22:24:35.820666 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.820658 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-2tnr9" event={"ID":"37e07277-b637-4e6f-b161-793af306f3ed","Type":"ContainerDied","Data":"a9afd7974fd7d9d534a6939a4a2a3f01e86f8bcd3a7b0aaa0e11ff3e9ae6093f"} Oct 30 22:24:35.820783 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.820680 4526 scope.go:122] "RemoveContainer" containerID="c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6" Oct 30 22:24:35.829414 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.829388 4526 scope.go:122] "RemoveContainer" containerID="c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6" Oct 30 22:24:35.829693 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:24:35.829675 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6\": container with ID starting with c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6 not found: ID does not exist" containerID="c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6" Oct 30 22:24:35.829754 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.829705 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6"} err="failed to get container status \"c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6\": rpc error: code = NotFound desc = could not find container \"c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6\": container with ID starting with c75ce9743d6adb15974d3ff6ad348e3c3e95f2b2c60e4ca6d8d1a8be783596e6 not found: ID does not exist" Oct 30 22:24:35.984536 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:35.984510 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-2tnr9"] Oct 30 22:24:36.030800 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:36.030773 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-2tnr9"] Oct 30 22:24:36.826767 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:36.826734 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" event={"ID":"05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7","Type":"ContainerStarted","Data":"ee4f3a5078005ba540dd14c4bb25945b9777c01a9a032e8d185908150b50be3b"} Oct 30 22:24:36.827297 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:36.826817 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:36.940434 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:36.940376 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" podStartSLOduration=2.128204373 podStartE2EDuration="2.940362174s" totalImagesPullingTime="812.157801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:36.939975775 +0000 UTC m=+4067.073348974" watchObservedRunningTime="2026-10-30 22:24:36.940362174 +0000 UTC m=+4067.073735372" Oct 30 22:24:37.983371 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:37.983337 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e07277-b637-4e6f-b161-793af306f3ed" path="/var/lib/kubelet/pods/37e07277-b637-4e6f-b161-793af306f3ed/volumes" Oct 30 22:24:42.477625 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:24:42.477587 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:24:42.838337 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:42.838302 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5644488b47-r9zp9" Oct 30 22:24:51.542741 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.542705 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6"] Oct 30 22:24:51.543166 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.543098 4526 cpu_manager.go:414] "containerMap: removing container" podUID="37e07277-b637-4e6f-b161-793af306f3ed" containerName="maas-api" Oct 30 22:24:51.543166 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.543109 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e07277-b637-4e6f-b161-793af306f3ed" containerName="maas-api" Oct 30 22:24:51.543241 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.543176 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e07277-b637-4e6f-b161-793af306f3ed" containerName="maas-api" Oct 30 22:24:51.546454 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.546439 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.598209 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.598175 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-daf0fd7e-serving-cert\"" Oct 30 22:24:51.649532 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.649500 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6"] Oct 30 22:24:51.657016 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.656989 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.657185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.657038 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwbf\" (UniqueName: \"kubernetes.io/projected/8d5f3a80-9794-4f62-9115-63ca83c12a7e-kube-api-access-5bwbf\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.657185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.657124 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f3a80-9794-4f62-9115-63ca83c12a7e-openshift-service-ca\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.657185 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.657181 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-metrics-certs\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.758836 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.758789 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.759052 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.758882 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwbf\" (UniqueName: \"kubernetes.io/projected/8d5f3a80-9794-4f62-9115-63ca83c12a7e-kube-api-access-5bwbf\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.759052 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.758917 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f3a80-9794-4f62-9115-63ca83c12a7e-openshift-service-ca\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.759052 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.758959 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-metrics-certs\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.759052 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:24:51.758966 4526 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-serving-cert: secret "maas-api-e2e-ait-daf0fd7e-serving-cert" not found Oct 30 22:24:51.759052 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:24:51.759046 4526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls podName:8d5f3a80-9794-4f62-9115-63ca83c12a7e nodeName:}" failed. No retries permitted until 2026-10-30 22:24:52.259024422 +0000 UTC m=+4082.392397615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls") pod "maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" (UID: "8d5f3a80-9794-4f62-9115-63ca83c12a7e") : secret "maas-api-e2e-ait-daf0fd7e-serving-cert" not found Oct 30 22:24:51.759698 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.759673 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f3a80-9794-4f62-9115-63ca83c12a7e-openshift-service-ca\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.761750 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.761728 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-metrics-certs\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:51.816693 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:51.816612 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwbf\" (UniqueName: \"kubernetes.io/projected/8d5f3a80-9794-4f62-9115-63ca83c12a7e-kube-api-access-5bwbf\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:52.263388 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:52.263305 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:52.265974 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:52.265952 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls\") pod \"maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:52.458576 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:52.458542 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:52.831470 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:52.831441 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6"] Oct 30 22:24:52.833809 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:24:52.833777 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5f3a80_9794_4f62_9115_63ca83c12a7e.slice/crio-1ff4f94a2932d13c87c3ef2345896ffa801715cc89af2752c361e2b08e08b9f2 WatchSource:0}: Error finding container 1ff4f94a2932d13c87c3ef2345896ffa801715cc89af2752c361e2b08e08b9f2: Status 404 returned error can't find the container with id 1ff4f94a2932d13c87c3ef2345896ffa801715cc89af2752c361e2b08e08b9f2 Oct 30 22:24:52.906312 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:52.906272 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" event={"ID":"8d5f3a80-9794-4f62-9115-63ca83c12a7e","Type":"ContainerStarted","Data":"1ff4f94a2932d13c87c3ef2345896ffa801715cc89af2752c361e2b08e08b9f2"} Oct 30 22:24:53.355579 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:53.355547 4526 scope.go:122] "RemoveContainer" containerID="51cc8c266cba3902f3ebf307f7e25f7e7565894249fb305e09687c14cef3cbd6" Oct 30 22:24:53.913001 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:53.912965 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" event={"ID":"8d5f3a80-9794-4f62-9115-63ca83c12a7e","Type":"ContainerStarted","Data":"211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba"} Oct 30 22:24:53.913425 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:53.913095 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:24:54.030023 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:54.029911 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" podStartSLOduration=2.175048443 podStartE2EDuration="3.029894823s" totalImagesPullingTime="854.84638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:54.028445489 +0000 UTC m=+4084.161818689" watchObservedRunningTime="2026-10-30 22:24:54.029894823 +0000 UTC m=+4084.163268061" Oct 30 22:24:59.923767 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:24:59.923739 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:25:12.477515 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:25:12.477464 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:25:42.478469 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:25:42.478425 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:25:49.243323 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.243292 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6"] Oct 30 22:25:49.243740 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.243528 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" podUID="8d5f3a80-9794-4f62-9115-63ca83c12a7e" containerName="maas-api" containerID="cri-o://211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba" gracePeriod=30 Oct 30 22:25:49.493212 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.493190 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:25:49.625138 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.625108 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5f3a80-9794-4f62-9115-63ca83c12a7e-kube-api-access-5bwbf\" (UniqueName: \"kubernetes.io/projected/8d5f3a80-9794-4f62-9115-63ca83c12a7e-kube-api-access-5bwbf\") pod \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " Oct 30 22:25:49.625350 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.625165 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls\") pod \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " Oct 30 22:25:49.625350 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.625253 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-metrics-certs\") pod \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " Oct 30 22:25:49.625350 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.625319 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5f3a80-9794-4f62-9115-63ca83c12a7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f3a80-9794-4f62-9115-63ca83c12a7e-openshift-service-ca\") pod \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\" (UID: \"8d5f3a80-9794-4f62-9115-63ca83c12a7e\") " Oct 30 22:25:49.625875 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.625816 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5f3a80-9794-4f62-9115-63ca83c12a7e-openshift-service-ca" pod "8d5f3a80-9794-4f62-9115-63ca83c12a7e" (UID: "8d5f3a80-9794-4f62-9115-63ca83c12a7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:49.627976 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.627943 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls" pod "8d5f3a80-9794-4f62-9115-63ca83c12a7e" (UID: "8d5f3a80-9794-4f62-9115-63ca83c12a7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:49.628114 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.627998 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5f3a80-9794-4f62-9115-63ca83c12a7e-kube-api-access-5bwbf" pod "8d5f3a80-9794-4f62-9115-63ca83c12a7e" (UID: "8d5f3a80-9794-4f62-9115-63ca83c12a7e"). InnerVolumeSpecName "kube-api-access-5bwbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:49.628114 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.628098 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-metrics-certs" pod "8d5f3a80-9794-4f62-9115-63ca83c12a7e" (UID: "8d5f3a80-9794-4f62-9115-63ca83c12a7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:49.726880 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.726834 4526 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-metrics-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:49.726880 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.726874 4526 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f3a80-9794-4f62-9115-63ca83c12a7e-openshift-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:49.726880 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.726887 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwbf\" (UniqueName: \"kubernetes.io/projected/8d5f3a80-9794-4f62-9115-63ca83c12a7e-kube-api-access-5bwbf\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:49.727142 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:49.726900 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5f3a80-9794-4f62-9115-63ca83c12a7e-maas-api-tls\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:50.047733 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.047696 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9"] Oct 30 22:25:50.049806 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.049779 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c905053-fsnb9"] Oct 30 22:25:50.177587 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.177550 4526 generic.go:356] "Generic (PLEG): container finished" podID="8d5f3a80-9794-4f62-9115-63ca83c12a7e" containerID="211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba" exitCode=0 Oct 30 22:25:50.177587 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.177589 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" Oct 30 22:25:50.177869 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.177644 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" event={"ID":"8d5f3a80-9794-4f62-9115-63ca83c12a7e","Type":"ContainerDied","Data":"211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba"} Oct 30 22:25:50.177869 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.177713 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6" event={"ID":"8d5f3a80-9794-4f62-9115-63ca83c12a7e","Type":"ContainerDied","Data":"1ff4f94a2932d13c87c3ef2345896ffa801715cc89af2752c361e2b08e08b9f2"} Oct 30 22:25:50.177869 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.177747 4526 scope.go:122] "RemoveContainer" containerID="211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba" Oct 30 22:25:50.188027 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.188006 4526 scope.go:122] "RemoveContainer" containerID="211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba" Oct 30 22:25:50.188316 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:25:50.188293 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba\": container with ID starting with 211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba not found: ID does not exist" containerID="211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba" Oct 30 22:25:50.188363 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.188328 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba"} err="failed to get container status \"211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba\": rpc error: code = NotFound desc = could not find container \"211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba\": container with ID starting with 211b7d79a12b03ee2a0b9210599d83db061c1566afc1413919fdfeed4ed9b7ba not found: ID does not exist" Oct 30 22:25:50.339936 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.339451 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6"] Oct 30 22:25:50.341098 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:50.341076 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-daf0fd7e-587c97f866-jgbw6"] Oct 30 22:25:51.983718 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:51.983686 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25171821-9a2a-434e-bf91-03c8ba9ab758" path="/var/lib/kubelet/pods/25171821-9a2a-434e-bf91-03c8ba9ab758/volumes" Oct 30 22:25:51.984169 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:25:51.984154 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5f3a80-9794-4f62-9115-63ca83c12a7e" path="/var/lib/kubelet/pods/8d5f3a80-9794-4f62-9115-63ca83c12a7e/volumes" Oct 30 22:26:12.478251 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:26:12.478209 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:26:17.153295 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.153257 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d"] Oct 30 22:26:17.153809 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.153626 4526 cpu_manager.go:414] "containerMap: removing container" podUID="8d5f3a80-9794-4f62-9115-63ca83c12a7e" containerName="maas-api" Oct 30 22:26:17.153809 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.153637 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5f3a80-9794-4f62-9115-63ca83c12a7e" containerName="maas-api" Oct 30 22:26:17.153809 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.153722 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5f3a80-9794-4f62-9115-63ca83c12a7e" containerName="maas-api" Oct 30 22:26:17.156793 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.156772 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.211006 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.210976 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-161f7a-serving-cert\"" Oct 30 22:26:17.260872 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.260841 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d"] Oct 30 22:26:17.284940 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.284903 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6sk\" (UniqueName: \"kubernetes.io/projected/d5ca315b-1255-467e-be49-5fdde147b3a7-kube-api-access-xj6sk\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.285105 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.284953 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.285105 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.285070 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.285231 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.285126 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ca315b-1255-467e-be49-5fdde147b3a7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.386170 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.386136 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6sk\" (UniqueName: \"kubernetes.io/projected/d5ca315b-1255-467e-be49-5fdde147b3a7-kube-api-access-xj6sk\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.386374 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.386183 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.386374 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.386223 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.386374 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.386254 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ca315b-1255-467e-be49-5fdde147b3a7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.386374 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:26:17.386349 4526 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-serving-cert: secret "maas-api-e2e-authcfg-http-161f7a-serving-cert" not found Oct 30 22:26:17.386658 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:26:17.386413 4526 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls podName:d5ca315b-1255-467e-be49-5fdde147b3a7 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:17.886392823 +0000 UTC m=+4168.019766000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls") pod "maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" (UID: "d5ca315b-1255-467e-be49-5fdde147b3a7") : secret "maas-api-e2e-authcfg-http-161f7a-serving-cert" not found Oct 30 22:26:17.386999 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.386972 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ca315b-1255-467e-be49-5fdde147b3a7-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.393037 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.393011 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-metrics-certs\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.443452 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.443382 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6sk\" (UniqueName: \"kubernetes.io/projected/d5ca315b-1255-467e-be49-5fdde147b3a7-kube-api-access-xj6sk\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.891736 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.891705 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:17.894286 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:17.894258 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:18.069406 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:18.069363 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:18.441523 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:18.441481 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d"] Oct 30 22:26:18.443780 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:26:18.443752 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ca315b_1255_467e_be49_5fdde147b3a7.slice/crio-31d3b9ec1c391e89c0c0c01197278626282fdaa257c67cc1757f9fcb226b2d8c WatchSource:0}: Error finding container 31d3b9ec1c391e89c0c0c01197278626282fdaa257c67cc1757f9fcb226b2d8c: Status 404 returned error can't find the container with id 31d3b9ec1c391e89c0c0c01197278626282fdaa257c67cc1757f9fcb226b2d8c Oct 30 22:26:19.314703 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:19.314671 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" event={"ID":"d5ca315b-1255-467e-be49-5fdde147b3a7","Type":"ContainerStarted","Data":"31d3b9ec1c391e89c0c0c01197278626282fdaa257c67cc1757f9fcb226b2d8c"} Oct 30 22:26:20.320811 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:20.320772 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" event={"ID":"d5ca315b-1255-467e-be49-5fdde147b3a7","Type":"ContainerStarted","Data":"0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce"} Oct 30 22:26:20.321252 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:20.320860 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:20.434282 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:20.434230 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" podStartSLOduration=2.574437201 podStartE2EDuration="3.434216228s" totalImagesPullingTime="859.779027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:20.4336781 +0000 UTC m=+4170.567051298" watchObservedRunningTime="2026-10-30 22:26:20.434216228 +0000 UTC m=+4170.567589420" Oct 30 22:26:26.330995 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:26.330966 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:26:42.477940 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:26:42.477856 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:26:53.401148 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:26:53.401110 4526 scope.go:122] "RemoveContainer" containerID="dd6d92687043fa7efe4fd3fd2835fc11823121ba22163cc7ec5f0f9d1f13f7bb" Oct 30 22:27:12.478200 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:27:12.478158 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:27:42.478364 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:27:42.478325 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:27:51.044043 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:27:51.044006 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr"] Oct 30 22:27:51.092935 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:27:51.092903 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-llvmr"] Oct 30 22:27:51.984705 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:27:51.984663 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96603be-4df0-45ca-98b2-92f599832c2c" path="/var/lib/kubelet/pods/c96603be-4df0-45ca-98b2-92f599832c2c/volumes" Oct 30 22:27:53.419860 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:27:53.419832 4526 scope.go:122] "RemoveContainer" containerID="01614c12a371090716d88382de091debd98f30b60ff38b1825fe0c9ee6e663e2" Oct 30 22:28:12.477335 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:28:12.477295 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:28:34.051676 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:34.051641 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm"] Oct 30 22:28:34.100838 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:34.100810 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gdtkm"] Oct 30 22:28:35.982941 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:35.982907 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083dad9d-22bc-424c-b89e-2eb451d75415" path="/var/lib/kubelet/pods/083dad9d-22bc-424c-b89e-2eb451d75415/volumes" Oct 30 22:28:36.166699 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:36.166657 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d"] Oct 30 22:28:36.964049 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:36.963989 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" podUID="d5ca315b-1255-467e-be49-5fdde147b3a7" containerName="maas-api" containerID="cri-o://0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce" gracePeriod=30 Oct 30 22:28:37.227170 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.227092 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:28:37.241217 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.241184 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ca315b-1255-467e-be49-5fdde147b3a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ca315b-1255-467e-be49-5fdde147b3a7-openshift-service-ca\") pod \"d5ca315b-1255-467e-be49-5fdde147b3a7\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " Oct 30 22:28:37.241711 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.241253 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls\") pod \"d5ca315b-1255-467e-be49-5fdde147b3a7\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " Oct 30 22:28:37.241711 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.241279 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ca315b-1255-467e-be49-5fdde147b3a7-kube-api-access-xj6sk\" (UniqueName: \"kubernetes.io/projected/d5ca315b-1255-467e-be49-5fdde147b3a7-kube-api-access-xj6sk\") pod \"d5ca315b-1255-467e-be49-5fdde147b3a7\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " Oct 30 22:28:37.241711 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.241310 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-metrics-certs\") pod \"d5ca315b-1255-467e-be49-5fdde147b3a7\" (UID: \"d5ca315b-1255-467e-be49-5fdde147b3a7\") " Oct 30 22:28:37.241711 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.241598 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ca315b-1255-467e-be49-5fdde147b3a7-openshift-service-ca" pod "d5ca315b-1255-467e-be49-5fdde147b3a7" (UID: "d5ca315b-1255-467e-be49-5fdde147b3a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:37.243574 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.243552 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls" pod "d5ca315b-1255-467e-be49-5fdde147b3a7" (UID: "d5ca315b-1255-467e-be49-5fdde147b3a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:37.243658 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.243599 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ca315b-1255-467e-be49-5fdde147b3a7-kube-api-access-xj6sk" pod "d5ca315b-1255-467e-be49-5fdde147b3a7" (UID: "d5ca315b-1255-467e-be49-5fdde147b3a7"). InnerVolumeSpecName "kube-api-access-xj6sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:37.243759 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.243743 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-metrics-certs" pod "d5ca315b-1255-467e-be49-5fdde147b3a7" (UID: "d5ca315b-1255-467e-be49-5fdde147b3a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:37.342482 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.342446 4526 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-maas-api-tls\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.342482 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.342478 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj6sk\" (UniqueName: \"kubernetes.io/projected/d5ca315b-1255-467e-be49-5fdde147b3a7-kube-api-access-xj6sk\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.342482 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.342516 4526 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ca315b-1255-467e-be49-5fdde147b3a7-metrics-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.342759 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.342526 4526 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ca315b-1255-467e-be49-5fdde147b3a7-openshift-service-ca\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:37.969852 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.969810 4526 generic.go:356] "Generic (PLEG): container finished" podID="d5ca315b-1255-467e-be49-5fdde147b3a7" containerID="0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce" exitCode=0 Oct 30 22:28:37.970055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.969882 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" Oct 30 22:28:37.970055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.969897 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" event={"ID":"d5ca315b-1255-467e-be49-5fdde147b3a7","Type":"ContainerDied","Data":"0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce"} Oct 30 22:28:37.970055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.969981 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d" event={"ID":"d5ca315b-1255-467e-be49-5fdde147b3a7","Type":"ContainerDied","Data":"31d3b9ec1c391e89c0c0c01197278626282fdaa257c67cc1757f9fcb226b2d8c"} Oct 30 22:28:37.970055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.970004 4526 scope.go:122] "RemoveContainer" containerID="0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce" Oct 30 22:28:37.979887 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.979868 4526 scope.go:122] "RemoveContainer" containerID="0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce" Oct 30 22:28:37.980230 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:28:37.980209 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce\": container with ID starting with 0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce not found: ID does not exist" containerID="0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce" Oct 30 22:28:37.980295 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:37.980241 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce"} err="failed to get container status \"0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce\": rpc error: code = NotFound desc = could not find container \"0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce\": container with ID starting with 0373dfc70c527ccbedf509400a31fddbf279c00c8fe9dc73af5292db44a6cbce not found: ID does not exist" Oct 30 22:28:38.134081 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:38.134039 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d"] Oct 30 22:28:38.138395 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:38.138360 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-161f7a-78f5d6996-dvd6d"] Oct 30 22:28:39.984784 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:39.984751 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ca315b-1255-467e-be49-5fdde147b3a7" path="/var/lib/kubelet/pods/d5ca315b-1255-467e-be49-5fdde147b3a7/volumes" Oct 30 22:28:42.477107 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:28:42.477070 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:28:53.452620 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:28:53.442420 4526 scope.go:122] "RemoveContainer" containerID="18084011b514ee6b6f8123bb2e67ef757ad01d57a59ead8fcd2a255961c83df3" Oct 30 22:29:06.046683 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:29:06.046630 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq"] Oct 30 22:29:06.095244 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:29:06.095203 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fe33fe5-q5dnq"] Oct 30 22:29:07.984441 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:29:07.984407 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa19f503-b85f-472d-b4aa-0de0c2279e3e" path="/var/lib/kubelet/pods/fa19f503-b85f-472d-b4aa-0de0c2279e3e/volumes" Oct 30 22:29:12.477597 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:29:12.477557 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:29:42.476866 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:29:42.476787 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:29:53.488080 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:29:53.488043 4526 scope.go:122] "RemoveContainer" containerID="5340ebc54aba5258fa957c02fc753684e2c8b21e61f96a26c7642bc17d3564a3" Oct 30 22:30:12.481727 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:30:12.481689 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:30:28.919834 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:30:28.919796 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-695b99ffdc-mbbg7_d44c216f-e8e9-4a71-93fc-c4081601a2b6/manager/0.log" Oct 30 22:30:31.069147 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:30:31.069097 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cnpp7_bd81cee0-0dc6-4f42-8a61-3b5780e223bf/server/0.log" Oct 30 22:30:32.742114 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:30:32.742077 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5644488b47-r9zp9_05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7/maas-api/0.log" Oct 30 22:30:42.477953 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:30:42.477916 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:30:53.507656 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:30:53.507620 4526 scope.go:122] "RemoveContainer" containerID="c5699f96f5faef1a5c9615e0674e349b106375a88af92f3f7f31b18b9e778452" Oct 30 22:30:56.237351 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:30:56.237322 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nsqsv_f50aabf8-5555-49b7-ac58-a6370c512b49/kuadrant-console-plugin/0.log" Oct 30 22:31:09.071256 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:09.071170 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-578b8bf98-fgttk_d5075246-d0cb-4e2b-af88-559679cf87bb/kube-auth-proxy/0.log" Oct 30 22:31:10.095683 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:10.095650 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-f7ssq_f9551403-60a7-4a70-a8e5-aa86043b3185/istio-proxy/0.log" Oct 30 22:31:12.477707 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:31:12.477671 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:31:20.895575 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:20.895540 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-nfl6x_d055332f-4f2b-4d52-842b-163859de6009/istio-proxy/0.log" Oct 30 22:31:21.187138 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:21.187060 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-qxjl4_e9630be5-74c9-4c96-aee6-459cc0c2943b/istio-proxy/0.log" Oct 30 22:31:22.207586 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:22.207554 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-r9sf9_1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de/payload-pre-processing/0.log" Oct 30 22:31:23.876637 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:23.876572 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-kk47r_d4f6ca98-2796-4b8a-acaf-ea95b126220e/main/0.log" Oct 30 22:31:24.567996 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:24.567957 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr_c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32/main/0.log" Oct 30 22:31:25.029518 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:25.029418 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z_3f6bfaff-39af-4c61-a3b5-5d25d0098a86/main/0.log" Oct 30 22:31:25.268011 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:31:25.267972 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq_c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b/main/0.log" Oct 30 22:31:42.477822 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:31:42.477775 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:32:05.597860 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:05.597826 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-695b99ffdc-mbbg7_d44c216f-e8e9-4a71-93fc-c4081601a2b6/manager/0.log" Oct 30 22:32:07.268326 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:07.268291 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cnpp7_bd81cee0-0dc6-4f42-8a61-3b5780e223bf/server/0.log" Oct 30 22:32:12.477223 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:32:12.477186 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:32:18.712182 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:18.712142 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5644488b47-r9zp9_05aa8ae1-8ebf-40eb-b8ba-dbb05b1f4ed7/maas-api/0.log" Oct 30 22:32:28.214003 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:28.213954 4526 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal" podUID="a4207178a950303c256d4f333088d96f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:32:41.901059 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:41.900974 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nsqsv_f50aabf8-5555-49b7-ac58-a6370c512b49/kuadrant-console-plugin/0.log" Oct 30 22:32:42.477578 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:32:42.477544 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:32:44.327308 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:44.327273 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-578b8bf98-fgttk_d5075246-d0cb-4e2b-af88-559679cf87bb/kube-auth-proxy/0.log" Oct 30 22:32:45.321375 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:45.321347 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-f7ssq_f9551403-60a7-4a70-a8e5-aa86043b3185/istio-proxy/0.log" Oct 30 22:32:46.072360 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:46.072328 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-nfl6x_d055332f-4f2b-4d52-842b-163859de6009/istio-proxy/0.log" Oct 30 22:32:56.320421 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:56.320390 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-qxjl4_e9630be5-74c9-4c96-aee6-459cc0c2943b/istio-proxy/0.log" Oct 30 22:32:57.407149 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:32:57.407112 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-r9sf9_1af9bcd6-6df5-4f62-88d6-c1fe98dbb2de/payload-pre-processing/0.log" Oct 30 22:33:12.477303 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:33:12.477268 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:33:18.832118 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:33:18.832083 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-kk47r_d4f6ca98-2796-4b8a-acaf-ea95b126220e/main/0.log" Oct 30 22:33:29.591718 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:33:29.591687 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8z7hnr_c4ee0ddf-4b0d-4a63-ad5b-a5dd6cc98a32/main/0.log" Oct 30 22:33:30.110881 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:33:30.110853 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gnw5z_3f6bfaff-39af-4c61-a3b5-5d25d0098a86/main/0.log" Oct 30 22:33:30.340318 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:33:30.340288 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qxvwq_c58e5a64-1b9f-4ba4-8f36-c7a59ccc066b/main/0.log" Oct 30 22:33:42.477141 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:33:42.477105 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:34:12.479043 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:34:12.478956 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:34:42.478989 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:34:42.478951 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:35:12.477795 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:35:12.477755 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:35:30.899892 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:30.899857 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4"] Oct 30 22:35:30.900393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:30.900046 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" containerID="cri-o://8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968" gracePeriod=30 Oct 30 22:35:30.949125 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:30.949095 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq"] Oct 30 22:35:30.949299 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:30.949279 4526 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" containerID="cri-o://8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3" gracePeriod=30 Oct 30 22:35:36.217939 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.217906 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:35:36.249346 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.249319 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:35:36.298775 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.298730 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-envoy\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.298775 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.298778 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-podinfo\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.298831 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-data\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.298864 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-kube-api-access-vf4fc\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-kube-api-access-vf4fc\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.298884 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-credential-socket\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.299055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.298907 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-data\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.299055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.298942 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9630be5-74c9-4c96-aee6-459cc0c2943b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9630be5-74c9-4c96-aee6-459cc0c2943b-istiod-ca-cert\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.298981 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-kube-api-access-6fn9k\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-kube-api-access-6fn9k\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.299055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299009 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-credential-socket\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299055 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299040 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-token\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-token\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299075 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-certs\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299106 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-socket\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299143 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9551403-60a7-4a70-a8e5-aa86043b3185-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9551403-60a7-4a70-a8e5-aa86043b3185-istiod-ca-cert\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299130 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-data" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299193 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-socket\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299230 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-podinfo\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299268 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-certs\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299276 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-credential-socket" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299317 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-envoy\") pod \"e9630be5-74c9-4c96-aee6-459cc0c2943b\" (UID: \"e9630be5-74c9-4c96-aee6-459cc0c2943b\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299347 4526 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-token\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-token\") pod \"f9551403-60a7-4a70-a8e5-aa86043b3185\" (UID: \"f9551403-60a7-4a70-a8e5-aa86043b3185\") " Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299396 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-credential-socket" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.299537 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299472 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9630be5-74c9-4c96-aee6-459cc0c2943b-istiod-ca-cert" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299777 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-data" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299789 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299812 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299829 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9630be5-74c9-4c96-aee6-459cc0c2943b-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299843 4526 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-credential-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.299913 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-certs" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.300063 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-socket" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.300216 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-socket" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.300393 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.300283 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9551403-60a7-4a70-a8e5-aa86043b3185-istiod-ca-cert" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:36.300928 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.300722 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-certs" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.301941 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.301907 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-envoy" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.302077 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.301980 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-podinfo" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:36.302284 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.302245 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-kube-api-access-vf4fc" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "kube-api-access-vf4fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.302428 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.302405 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-token" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.302693 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.302664 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-kube-api-access-6fn9k" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "kube-api-access-6fn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.302802 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.302718 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-envoy" pod "e9630be5-74c9-4c96-aee6-459cc0c2943b" (UID: "e9630be5-74c9-4c96-aee6-459cc0c2943b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.303108 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.303091 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-podinfo" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:36.303240 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.303224 4526 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-token" pod "f9551403-60a7-4a70-a8e5-aa86043b3185" (UID: "f9551403-60a7-4a70-a8e5-aa86043b3185"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.400460 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400425 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf4fc\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-kube-api-access-vf4fc\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400460 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400454 4526 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-data\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400460 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400465 4526 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fn9k\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-kube-api-access-6fn9k\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400476 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400508 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400517 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400525 4526 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9551403-60a7-4a70-a8e5-aa86043b3185-istiod-ca-cert\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400534 4526 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-workload-socket\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400543 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400551 4526 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-workload-certs\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400558 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400566 4526 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-token\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400575 4526 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9551403-60a7-4a70-a8e5-aa86043b3185-istio-envoy\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.400765 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.400583 4526 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9630be5-74c9-4c96-aee6-459cc0c2943b-istio-podinfo\") on node \"ip-10-0-128-26.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.879613 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.879580 4526 generic.go:356] "Generic (PLEG): container finished" podID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerID="8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968" exitCode=0 Oct 30 22:35:36.879832 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.879626 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" Oct 30 22:35:36.879832 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.879667 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" event={"ID":"e9630be5-74c9-4c96-aee6-459cc0c2943b","Type":"ContainerDied","Data":"8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968"} Oct 30 22:35:36.879832 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.879721 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4" event={"ID":"e9630be5-74c9-4c96-aee6-459cc0c2943b","Type":"ContainerDied","Data":"c25819e6cbfd06a23a5c6c13e8dee09c24e3371301ed7ee3e60fcfc769930a6f"} Oct 30 22:35:36.879832 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.879749 4526 scope.go:122] "RemoveContainer" containerID="8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968" Oct 30 22:35:36.881098 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.881067 4526 generic.go:356] "Generic (PLEG): container finished" podID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerID="8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3" exitCode=0 Oct 30 22:35:36.881245 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.881113 4526 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" Oct 30 22:35:36.881245 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.881151 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" event={"ID":"f9551403-60a7-4a70-a8e5-aa86043b3185","Type":"ContainerDied","Data":"8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3"} Oct 30 22:35:36.881245 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.881199 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq" event={"ID":"f9551403-60a7-4a70-a8e5-aa86043b3185","Type":"ContainerDied","Data":"2aa7a709f6e3debddbaa017c2d73f1aa501556704bfffa70480648bef08ad796"} Oct 30 22:35:36.890656 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.890637 4526 scope.go:122] "RemoveContainer" containerID="8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968" Oct 30 22:35:36.891004 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:35:36.890932 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968\": container with ID starting with 8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968 not found: ID does not exist" containerID="8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968" Oct 30 22:35:36.891004 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.890968 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968"} err="failed to get container status \"8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968\": rpc error: code = NotFound desc = could not find container \"8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968\": container with ID starting with 8c0b1621525a0636c1068852c67ba20737e8f07c14a4a4cd21e2a7c8c4c87968 not found: ID does not exist" Oct 30 22:35:36.891004 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.890993 4526 scope.go:122] "RemoveContainer" containerID="8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3" Oct 30 22:35:36.900043 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.900023 4526 scope.go:122] "RemoveContainer" containerID="8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3" Oct 30 22:35:36.900308 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:35:36.900287 4526 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3\": container with ID starting with 8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3 not found: ID does not exist" containerID="8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3" Oct 30 22:35:36.900357 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:36.900319 4526 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3"} err="failed to get container status \"8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3\": rpc error: code = NotFound desc = could not find container \"8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3\": container with ID starting with 8fa4e4d7adfb272d65a4e5f7ce3154b1dd836a07fa81a30e62f221eeef05e6b3 not found: ID does not exist" Oct 30 22:35:37.043772 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:37.043734 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4"] Oct 30 22:35:37.045272 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:37.045253 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qxjl4"] Oct 30 22:35:37.154985 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:37.154899 4526 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq"] Oct 30 22:35:37.159726 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:37.159694 4526 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-f7ssq"] Oct 30 22:35:37.984327 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:37.984296 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" path="/var/lib/kubelet/pods/e9630be5-74c9-4c96-aee6-459cc0c2943b/volumes" Oct 30 22:35:37.984801 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:35:37.984772 4526 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" path="/var/lib/kubelet/pods/f9551403-60a7-4a70-a8e5-aa86043b3185/volumes" Oct 30 22:35:42.476851 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:35:42.476804 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:36:12.477182 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:36:12.477146 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:36:42.477662 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:36:42.477622 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:37:12.477139 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:37:12.477058 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:37:42.477181 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:37:42.477145 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:37:43.978687 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:37:43.978656 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bbwl8_db28a65b-b841-4446-b062-9526c7e8139a/global-pull-secret-syncer/0.log" Oct 30 22:37:46.677916 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:37:46.677869 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tljvj_62b9feff-59c5-45d5-a398-c78ad8dd590b/konnectivity-agent/0.log" Oct 30 22:37:47.220939 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:37:47.220912 4526 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-26.us-west-1.compute.internal_a4207178a950303c256d4f333088d96f/haproxy/0.log" Oct 30 22:38:12.477731 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:38:12.477692 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor" Oct 30 22:38:31.147774 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.147738 4526 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4"] Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148106 4526 cpu_manager.go:414] "containerMap: removing container" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148116 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148136 4526 cpu_manager.go:414] "containerMap: removing container" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148141 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148163 4526 cpu_manager.go:414] "containerMap: removing container" podUID="d5ca315b-1255-467e-be49-5fdde147b3a7" containerName="maas-api" Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148169 4526 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ca315b-1255-467e-be49-5fdde147b3a7" containerName="maas-api" Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148231 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9551403-60a7-4a70-a8e5-aa86043b3185" containerName="istio-proxy" Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148240 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9630be5-74c9-4c96-aee6-459cc0c2943b" containerName="istio-proxy" Oct 30 22:38:31.148267 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.148248 4526 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ca315b-1255-467e-be49-5fdde147b3a7" containerName="maas-api" Oct 30 22:38:31.151230 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.151213 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.202811 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.202784 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tlnnt\"/\"openshift-service-ca.crt\"" Oct 30 22:38:31.202974 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.202818 4526 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tlnnt\"/\"kube-root-ca.crt\"" Oct 30 22:38:31.213064 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.213041 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-proc\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.213182 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.213075 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-sys\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.213240 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.213217 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-podres\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.213299 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.213286 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshxf\" (UniqueName: \"kubernetes.io/projected/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-kube-api-access-tshxf\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.213335 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.213315 4526 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-lib-modules\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.252408 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.252375 4526 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tlnnt\"/\"default-dockercfg-w9mvk\"" Oct 30 22:38:31.254070 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.254045 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4"] Oct 30 22:38:31.314434 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314398 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-lib-modules\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.314654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314462 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-proc\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.314654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314511 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-sys\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.314654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314546 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-podres\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.314654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314580 4526 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshxf\" (UniqueName: \"kubernetes.io/projected/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-kube-api-access-tshxf\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.314654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314587 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-proc\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.314654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314599 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-sys\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.314654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314625 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-podres\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.314654 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.314578 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-lib-modules\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.371326 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.371295 4526 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshxf\" (UniqueName: \"kubernetes.io/projected/c1ae14ca-6f18-47c1-b5f8-3f87804fac2a-kube-api-access-tshxf\") pod \"perf-node-gather-daemonset-srmq4\" (UID: \"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.463921 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.463838 4526 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:31.836622 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:31.836598 4526 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4"] Oct 30 22:38:31.839425 ip-10-0-128-26 kubenswrapper[4526]: W1030 22:38:31.839396 4526 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc1ae14ca_6f18_47c1_b5f8_3f87804fac2a.slice/crio-438b6877d660b0edd5fd6c2457082c1c898a0b37c8ed9b17e06122b5e04547a1 WatchSource:0}: Error finding container 438b6877d660b0edd5fd6c2457082c1c898a0b37c8ed9b17e06122b5e04547a1: Status 404 returned error can't find the container with id 438b6877d660b0edd5fd6c2457082c1c898a0b37c8ed9b17e06122b5e04547a1 Oct 30 22:38:32.626760 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:32.626719 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" event={"ID":"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a","Type":"ContainerStarted","Data":"f54cb2405cf2385cb2bf24cab715fff07df6286a34fbd8a235a28d1cc01c34ea"} Oct 30 22:38:32.627211 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:32.626781 4526 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" event={"ID":"c1ae14ca-6f18-47c1-b5f8-3f87804fac2a","Type":"ContainerStarted","Data":"438b6877d660b0edd5fd6c2457082c1c898a0b37c8ed9b17e06122b5e04547a1"} Oct 30 22:38:32.627211 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:32.626869 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:32.742719 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:32.742665 4526 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" podStartSLOduration=1.74265213 podStartE2EDuration="1.74265213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:32.740973759 +0000 UTC m=+4902.874346958" watchObservedRunningTime="2026-10-30 22:38:32.74265213 +0000 UTC m=+4902.876025327" Oct 30 22:38:38.640936 ip-10-0-128-26 kubenswrapper[4526]: I1030 22:38:38.640904 4526 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-srmq4" Oct 30 22:38:42.477127 ip-10-0-128-26 kubenswrapper[4526]: E1030 22:38:42.477083 4526 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4z7zj" containerName="extractor"